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in  their  training  program  development  efforts  and  to  study  the  utility 
of  the  6.2  ISO  model  in  an  application  to  an  existing  aviation  system. 

The  latter  goal  included  gathering  information  on  the  strengths  and 
weaknesses  of  the  model,  making  recommendations  for  modifications  to 
the  model,  and  collecting  resource  utilization  data.  Input  data  came 
from  a validated  list  of  pilot  and  RIO  tasks  which  was  furnished  by 
the  government.  Tasks  from  this  list  were  analyzed  into  a hierarchical 
structure  of  supporting  skills  and  knowledge.  This  analysis  formed 
the  basis  for  the  development  of  instructional  objectives.  Preferred 
and  alternate  media  were  selected  for  each  of  the  objectives.  Objectives 
were  then  grouped  into  lessons  and  the  lessons  sequenced  to  form  the 
pilot  and  RIO  syllabi.  Finally,  a training  support  requirements  analysis 
was  performed  to  estimate  resource  requirements  for  development,  imple- 
mentation, revision,  and  maintenance  of  the  two  training  courses. 

Problems  encountered  during  this  program  were  discussed  and  recommen- 
dations for  changes  to  the  ISD  model  were  presented,  t 
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SUMMARY 

This  report  covers  the  Phase  I activities  of  a project 
for  development  of  ISD-based  training  programs  for  Marine  and 
Navy  F-4  pilots  and  Radar  Intercept  Officers  (RIO).  In  addition 
to  developing  specified  end-products  to  be  used  in  subsequent 
ISO  phases,  it  was  an  effort  to  evaluate  the  utility  of  the 
emerging  6.2  ISO  model  by  identifying  its  strengths  and  weak- 
nesses. Project  activities  were  conducted  on-site  at  MCAS,  Yuma 
with  government- furnished  subject  matter  experts  providing  input 
to  major  tasks  under  the  direction  of  Allen  Corporation  ISD 
personnel . 

Major  tasks  completed  during  this  project  were  objectives 
hierarchy  development,  media  selection,  syllabus  development,  and 
training  support  requirements  analysis.  Each  of  these  activities 
is  summarized  below. 

OBJECTIVES  HIERARCHY  DEVELOPMENT 

A job  analysis  was  conducted  prior  to  contract  award  to 
Allen  Corporation.  This  survey  provided  a listing  of  tasks 
selected  for  training.  These  tasks  were  the  input  to  objectives 
hierarchy  development.  During  hierarchy  development,  tasks  were 
analyzed  to  identify  subtasks  which  were  components  of  the  major 
tasks.  Behavioral  objectives  were  developed  for  each  subtask 
and,  along  with  the  major  tasks,  were  arranged  in  hierarchy 
formats  to  show  their  interrelationships,  i.e.  superordinate, 
coordinate,  and  subordinate  relationships. 

MEDIA  SELECTION 

Preferred  and  alternate  instructional  media  were  selected 
for  each  objective  identified  during  hierarchy  development. 
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Selections  were  made  on  the  basis  of  an  algorithm  in  which 
the  following  variables  were  considered:  (1)  level  of  expected 
student  behavior,  (2)  level  of  content  taught,  (3)  amount  of 
practice  required  for  mastery,  (4)  minimum  display  requirements, 
and  (5)  amount  of  memorization  required.  During  the  next  step, 
syllabus  development,  media  selections  for  the  objectives 
were  used  to  determine  the  preferred  medium  for  each  lesson. 

SYLLABUS  DEVELOPMENT 

Objectives  were  grouped  to  form  lessons.  First,  flight 
lessons  were  formulated  using  objectives  selected  for  flights 
during  media  selection.  Next,  objectives  requiring  simulators 
were  grouped  to  form  lessons  to  support  the  previously  formed 
flight  lessons.  Finally,  ground  school  lessons  were  structured 
to  support  the  simulator  and  flight  lessons.  After  the  ground 
school  objectives  were  grouped  into  lessons,  media  selections 
were  reviewed  to  determine  if  media  for  all  objectives  in  a 
lesson  were  the  same.  If  not,  adjustments  in  media  and/or 
lesson  content  were  made  to  select  the  preferred  medium  for 
each  lesson. 

The  resultant  syllabus  embodied  an  individualized  instruction- 
al system  in  which  the  predominant  medium  was  slide-tape  presen- 
tation. Other  media  included  workbooks,  videotapes,  simulators, 
and  the  aircraft.  Lessons  were  sequenced  to  facilitate  acquisition 
and  retention  by  proceeding  from  ground  school  instruction  to 
hands-on  practice  in  a simulator,  to  practice  in  the  aircraft. 

TRAINING  SUPPORT  REQUIREMENTS  ANALYSIS 


The  final  task  of  this  project  was  to  estimate  cost, 
facilities,  and  personnel  requirements  to  develop,  implement, 
revise  and  maintain  the  training  program  outlined  in  the 
syllabus.  Development  cost  estimates  were  based  on  a survey  of 
commercial  and  government  organizations.  This  survey  yielded 
costs  and  personnel  requirements  for  lessons  in  each  of  the 
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selected  media.  Similar  requirements  for  revision  of  materials 
were  also  determined.  Facilities  requirements  included  that 
required  by  materials  development  personnel  and  for  implementation 
of  the  training  course. 

CONCLUSIONS 

The  general  conclusion  of  this  project  was  that  the  ISD 
model  used  provides  a viable  approach  to  ISD  implementation. 

A number  of  problems  were  encountered,  which  resulted  in  suggestions 
for  modifications  to  the  model.  The  experience  gained  during  this 
project  and  the  resultant  recommendations  will  assist  in  the 
further  refinement  of  the  6.2  model. 


PREFACE 

The  Naval  Training  Equipment  Center  has  a continuing 
interest  in  the  use  of  ISD  techniques  to  upgrade  aircrew 
training  programs  and  in  the  process  to  develop  and  refine 
a model  for  future  ISD  programs.  This  F-4  program  followed 
four  similar  ISD  programs  (SH-2F  LAMPS,  A-6E  TRAM,  EA-6B  ICAP, 
and  E-2C  HAWKEYE)  in  which  different  ISD  methodologies  were 
used  by  the  contractors  involved.  As  a result  of  these  pro- 
grams, a standard  ISD  model  was  developed.  The  F-4  program  is 
the  first  program  in  which  the  model  has  been  implemented  by 
a contractor  other  than  those  involved  in  the  previous  ISD 
programs.  The  work  was  performed  by  the  Allen  Corporation 
of  America  under  Contract  N61339-77-C-0081 . A multi -phase 
ISD  effort  was  planned.  The  Phase  One  analysis  was  completed 
and  is  described  in  this  report. 

The  operational  goal  of  this  project  was  to  develop  the 
framework  for  pilot  and  RIO  training  programs,  which  could 
be  expanded  into  instructional  materials  for  implementation 


in  subsequent  phases.  The  research  and  development  goals 
were  to:  (a)  evaluate  how  well  an  independent  contractor,  using 
the  detailed  ISD  specification  and  associated  Data  Item 
Descriptions,  could  implement  the  model,  (b)  identify  strengths 
and  weaknesses  of  the  model,  (c)  obtain  recommendations  for 
modifications  to  the  model  and  (d)  acquire  cost,  scheduling, 
and  manpower  data  for  future  ISD  planning. 
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SECTION  I 
INTRODUCTION 


BACKGROUND 

This  project  is  the  latest  in  a series  of  programs  sponsored 
by  the  Naval  Training  Equipment  Center  (NAVTRAEQUIPCEN)  to  develop, 
refine,  and  standardize  techniques  used  in  Instructional  Systems 
Development  (ISO).  The  goal  of  these  programs  is  to  define  an  ISD 
model  which  will  provide  a standardized  approach  to  developing 
training  programs  for  operators  and  maintenance  personnel  and  at 
the  same  time  to  upgrade  the  quality  of  aircrew  training  on 
selected  aviation  systems. 

The  first  four  ISD  programs  in  the  series  were  carried  out 
simultaneously  by  different  contractors  under  broad  ISD  guidelines 
specified  by  NAVTRAEQUIPCEN.  The  purpose  of  providing  only  broad 
guidelines  for  discrete  processes  and  end  products  was  to  allow 
the  different  contractors  flexibility  to  utilize  their  own  tech- 
niques and  procedures.  Under  the  four  contracts  the  analysis 
steps  in  the  ISD  process  up  through  lesson  specifications  were 
carried  out  for  four  different  aircrew  training  courses.  The 
aircrewmen  for  which  the  ISD  process  was  initiated  were  the  pilot, 
NFOs,  and  FT  of  the  E-2C  Hawkeye,  the  pilot  and  ECMOs  of  the 
EA-6B  Prowler,  the  pilot  and  sensor  operator  of  the  SH-2F  LAMPS, 
and  the  pilot  and  NFO  of  the  A-6E  TRAM. 

Each  of  the  four  programs  required  the  development  of 
training  programs  for  two  different  types  of  crewmembers:  (1) 
pilot  training  which  has  a large  manipulative  skills  component 
supplemented  with  cognitive  instruction;  and  (2)  NFO  training, 
which  primarily  involves  instruction  in  interpretation  and  decision 
making  processes.  The  programs  also  encompassed  a wide  range  of 
aircraft  types  including  an  attack  aircraft  (A-6E),  an  electronic 
countermeasures  aircraft  (EA-6B),  an  airborne  early  warning  and 
surveillance  aircraft  (E-2C),  and  an  anti-submarine  helicopter 
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(SH-2F).  These  four  programs  with  their  variety  of  aircrew  types 
and  aircraft  missions  provided  the  information  from  which  a basic 
Navy  model  for  ISD  was  developed.  In  addition,  a large  amount  of 
data  on  the  manpower  and  costs  required  to  do  large  ISD  programs 
was  collected. 

The  second  step  in  the  series  of  ISD  programs  was  the 
formulation  of  a model  using  as  a data  base  the  procedures  employed 
by  the  different  contractors  in  the  four  previous  programs.  The 
purpose  was  to  analyze  the  strengths  and  weaknesses  of  the  different 
approaches  to  ISD  and  to  synthesize  the  strengths  of  the  models  into 
one  model  which  would  become  the  Navy  standard  for  training  program 
development.  The  use  of  the  model  in  future  programs  would  facilitate 
management  by  contract  technical  monitors  and  would  insure  a uniformly 
high  quality  product.  The  resultant  model  has  been  referred  to  as 
the  "6.2  Model".  The  most  recent  version  of  the  model  is  delineated 
in  Military  Specification  MIL-T-29053,  dated  19  October  1977,  and 
is  entitled  "Training  Requirements  for  Aviation  Weapon  Systems". 

The  F-4  program  which  is  the  most  recent  in  the  series  of 
ISD  programs,  emerged  as  the  result  of  two  needs.  One  was  the 
recognized  shortcomings  of  the  F-4  pilot  and  Radar  Intercept 
Officer  (RIO)  training  courses.  The  second  was  the  need  to  evaluate 
how  effectively  a contractor,  not  involved  in  the  earlier  ISD 
programs,  could  apply  the  "6.2  Model"  to  an  existing  aircrew 
training  program.  Related  to  this  second  need  was  the  continuing 
requirement  to  collect  data  on  ISD  manpower  and  cost  requirements. 

Specific  deficiencies  in  the  two  F-4  training  courses  were 
identified  in  the  problem  analysis.  It  was  noted  that  objective 
performance  standards  and  standardized  criteria  and  evaluation 
methods  have  not  been  defined  for  academic,  simulator,  or  flight 
training.  This  lack  of  standardization  and  specific  evaluation 
criteria  significantly  inhibits  efforts  to  improve  the  training. 
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Furthermore,  an  extremely  low  wash-out  rate  suggests  that  the  exit 
level  skills  of  the  trainees  are  not  being  assessed  with  adequate 
precision.  Hence,  optimal  remedial  action,  appropriate  to  students 
and/or  the  training  program  is  hampered.  It  was  also  noted  that 
the  capabilities  of  existing  training  devices  are  constraining  to 
the  current  training  programs. 

The  selection  of  a contractor,  familiar  with  ISO  procedures 
but  without  specific  experience  in  the  application  of  the  model, 
was  intended  to  serve  two  purposes:  (1)  an  evaluation  of  the 
clarity  and  utility  of  the  Data  Item  Descriptions  (DIDs)  and  the 
specification,  i.e.  determining  how  accurately  the  model  could  be 
interpreted  and  implemented  by  the  contractor;  and  (2)  a source  of 
recommendations  for  modifications  that  will  improve  the  procedures 
in  future  ISD  programs. 

The  analysis  phase  of  the  F-4  ISD  program,  which  is  discussed 
in  this  report,  was  applied  in  that  the  end  products  constituted 
the  first  steps  in  the  process  of  developing  an  ISD-based  training 
program  and  will  be  used  as  the  basis  upon  which  the  design  and 
development  phases  will  be  formulated.  It  was  research  in  that  it 
was  the  first  application  of  a rigidly  specified  model  by  a con- 
tractor other  than  those  who  had  developed  the  steps  and  procedures 
which  were  incorporated  into  the  model. 

STATEMENT  OF  THE  PROBLEM 


The  primary  goals  of  the  analysis  phase  of  the  F-4  ISD  program 
were  to  implement  the  ISD  model  in  accordance  with  Specification 
N215-264  and  to  prepare  and  deliver  the  specified  end  products  in 
accordance  with  applicable  DIDs.  Analyses  were  performed  for  both 
the  Navy  and  Marine  training  courses  and  the  F-4  J and  N models. 

The  starting  point  for  this  contract  was  the  task  of  developing 
objectives  hierarchies  and  Specific  Behavioral  Objectives  (SBOs) 
from  the  Task  Listing.  This  listing,  which  was  developed  by  the 
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Marine  ISD  team  at  MCAS,  Yuma  under  the  guidance  of  Courseware, 

Inc.  was  supplied  to  Allen  Corporation  at  the  beginning  of  the 
contract. 

The  procedure  specified  by  NAVTRAEQUIPCEN  was  to  have  Marine 
and  Navy  F-4  pilot  and  RIO  subject  matter  experts  (SMEs)  carry  out 
those  tasks  in  the  ISD  process  which  required  knowledge  of  F-4 
operations,  tactics,  and  procedur  This  was  done  under  the 
technical  guidance  of  Allen  C-'p’oi'/^S:. " «tfl*ks  in  which 

expert  knowledge  was  not  required  were  carried  out  primarily  by 
Allen  Corporation  personnel.  The  major  tasks  for  which  deliverable 
end  products  were  specified  were: 

1.  Objectives  Hierarchy  and  SBO  Development 

2.  Media  Selection 

3.  Syllabus  Development 

4.  Training  Support  Requirements  Analysis  (TSRA) 

SYSTEM  DESCRIPTION 

THE  AIRCRAFT.  The  F-4  is  a two-place,  supersonic,  long-range,  all- 
weather  fighter  built  by  McDonnell  Douglas  Corporation.  It  is  de- 
signed for  intermediate  and  long  range  high  altitude  interceptions 
using  missiles  as  the  principal  armament  and  for  intermediate  or 
long  range  attack  missions  to  deliver  airborne  weapons/stores.  The 
aircraft  is  powered  by  either  two  single  rotor,  axial  flow,  variable 
stator  turbojet  J79-GE-8  or  J79-GE-10  engines  with  afterburners. 

It  is  capable  of  carrier  operations. 

The  degree  of  sophistication  in  modern  weapons  systems  has 
necessitated  the  employment  of  two-man  crews  in  many  of  the  current 
fighters'  to  minimize  the  probability  of  operator  overload.  Although 
there  is  a clear  division  of  labor  between  the  pilot  and  RIO  duties, 
they  must  function  as  an  efficient  team  to  carry  out  the  missions 
of  the  F-4.  The  training  program  for  both  crew  positions  must, 
therefore,  stress  individual  duties,  as  well  as,  contain  a strong 
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emphasis  upon  the  teamwork  required.  The  most  successful  fighter 
teams  are  those  who  work  together  continuously  and  know  each 
other's  reactions,  strengths  and  weaknesses. 

THE  MISSIONS.  The  F-4  has  two  primary  missions,  air-to-air 
Interceptions  and  air-to-ground  delivery  of  airborne  weapons/stores. 

In  addition,  the  F-4  performs  armed  escort  and  reconnaissance  missions. 
The  primary  weapons  systems  used  to  carry  out  these  missions  are  the 
bombing  equipment,  the  data  link  system,  the  direct  radar  scope 
camera,  electronic  countermeasure  equipment,  gunnery  equipment, 
and  the  missile  control  equipment.  Since  the  F-4  has  been  in  the 
Navy/Marine  Inventory  for  a number  of  years,  a large  amount  of 
experience  in  executing  the  above  missions  in  combat,  practice,  and 
training  environments  has  been  accumulated. 

THE  TRAINING  PROGRAM.  The  existing  Navy  and  Marine  transition 
courses  are  the  results  of  many  iterations  and  refinements  over 
the  years  since  the  F-4  was  introduced.  The  Marine  syllabus  in- 
cludes training  to  60  percent  (combat  capable)  with  the  remainder  of 
training  to  100  percent  proficiency  (fully  combat  qualified)  taking 
place  In  tactical  group  training  at  operational  squadrons.  Graduates 
of  the  Navy  syllabus  are  80  percent  proficient  with  20  percent  left 
for  tactical  group  training  at  operational  squadrons.  A summary  of 
the  total  training  for  Marine  F-4  pilots  and  RIOs  is  shown  in  Table  1. 
Included  is  the  training  to  60  percent  proficiency  at  MCAS,  Yuma, 
plus  the  additional  training  provided  at  duty  squadrons.  Table  2 
provides  a further  breakdown  of  ground  school  training  which  currently 
takes  place  at  VMFAT-101,  MCAS,  Yuma.  Tables  3 and  4 are  break- 
downs by  .sorties/hours  and  simulator  periods/hours  of  the  Combat 
Capable  Training  portions  of  the  pilot  and  RIO  syllabi. 

Trainees  for  the  pilot  transition  course  come  from  four 
Advanced  Jet  Training  programs.  RIO  trainees  come  from  the 
Advanced  Naval  Flight  Officer  school.  Students  for  the  refresher 
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TABLE  1.  SUMMARY  OF  BASIC  MARINE  TRANSITION  AND  CONVERSION 
TRAINING  FOR  F-4  PILOTS  and  RIOs 


Weeks 

Course 

Activity 

1-3 

Ground  school  and  flight  simulator 
training 

Training  group* 

4-6 

Air-to-air  and  air-to-ground 
courses 

MAWTU* 

7-26 

Combat  capable  training 

Training  group* 

27-38 

Combat  ready  training 

Tactical  group** 

39-46 

Combat  qualification  training 

Tactical  group** 

47-54 

Full  combat  qualification  training 

Tactical  group** 

* Takes  place  at  FRS 

**  Takes  place  at  operational  squadron 
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TABLE  2.  BREAKDOWN  OF  TRANSITION  GROUND  SCHOOL  AND 
SIMULATOR/ FLIGHT  TRAINING  AT  MCAS,  YUMA 


Course 


Duration 


Ground  School  2 weeks 

Aircraft  Systems 
Cross-Country  Packets 
Aircraft  Preflight 
Plane  Captain  Signals 
Safety 

Aerodynamics 
Emergency  Procedures 
Course  Rules 

Aircrew  Teamwork  and  Responsibilities 

Basic  Radar 

Navigation  Computer 

Airborne  Missile  Control  System 

Flight  Planning 

Performance  Data 

Flight/Simulator  Training  1 week 
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TABLE  3.  BREAKDOWN  OF  THE  COMBAT  CAPABLE  TRAINING  PORTION 
OF  THE  PILOT  SYLLABUS 


FLIGHT  SIM.  TOTAL 


STAGE 

SORTIES/HOURS 

1 

PERIODS/HRS 

TRAINING 

Basic  Qualification 

Entry  level  skills 

Familiarization 

4/4.8 

8/12.0 

12/16.8 

Instrument 

4/6.0 

- 

4/  6.0 

Formation 

3/3.9 

- 

3/  3.9 

FMLP 

1/0.8 

- 

1/  0.8 

All  Weather  Intercept 

7/8.4 

12/18.0 

19/26.4 

Missile  Fire 

1/1.5 

1/  1.5 

2/  3.0 

Aerial  Ref. 

2/2.6 

- 

2/  2.6 

Ground  Attack 

9/9.0 

- 

9/  9.0 

Vis.  Nav. 

1/1.3 

- 

1/  1.3 

BAM 

6/6.0 

1/  1.5 

7/  7.5 

Fighter  Intercept 

2/2.4 

2/  3.0 

4/  5.4 

Fighter  Weapons 

7/5.6 

- 

7/  5.6 

47/52.3 

24/36.0 

71/88.3 
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TABLE  4.  BREAKDOWN  OF  THE  COMBAT  CAPABLE  TRAINING  PORTION 


OF  THE  RIO  SYLLABUS 

FLIGHT 

SIM. 

TOTAL 

STAGE 

SORTIES/HOURS  PERIODS/HRS 

TRAINING 

Basic  Qualification 

Entry  level  skills 

Familiarization 

1/  1.2 

8/12.0 

9/13.2 

Instrument 

5/  7.5 

- 

5/  7.5 

Formation 

1/  1.3 

- 

1/  1.3 

FMLP 

1/  0.8 

- 

1/  0.8 

All  Weather  Intercept 

10/12.0 

12/18.0 

22/30.0 

Missile  Fire 

1/  1.5 

1/  1.5 

2/  3.0 

Aerial  Ref. 

2/  2.6 

- 

2/  2.6 

Ground  Attack 

9/  9.0 

- 

9/  9.0 

Vis.  Nav. 

1/  1.3 

- 

1/  1.3 

BAM 

5/  5.0 

1/  1.5 

6/  6.5 

Fighter  Intercept 

2/  2.4 

2/  4.0 

4/  6.4 

Fighter  Weapons 

7/  5.6 

- 

7/  5.6 

45/50.2 

23/31.0 

68/87.2 
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pilot  and  RIO  courses  fall  into  two  categories,  those  who  have 
not  flown  in  an  F-4  for  6-18  months  and  those  who  have  not  flown 
in  an  F-4  for  more  than  18  months.  The  training  which  these 
pilots  and  RIOs  receive  is  a subset  of  the  total  transition 
course.  Instructors  Under  Training  (IUT)  receive  a special  six- 
week  IUT  course  in  addition  to  refresher  ground  school  and 
simulator  training. 

During  transition  training  at  MCAS,  Yuma,  trainees  are 
assigned  collateral  duties  which  require  an  average  of  two  hours 
per  day.  These  jobs  include  assignments  in  operations,  adminis- 
tration, maintenance,  intelligence,  and  logistics  and  embarkation. 
Other  duties  include  operations  duty  officer,  squadron  duty  officer, 
maintenance  duty  officer,  and  legal  investigations.  Due  to  these 
collateral  duties  trainees  are  not  expected  to  put  in  a "full 
training  day".  In  the  day-by-day  schedule  produced  as  part  of  the 
F-4  ISD  program  allowances  are  made  for  these  collateral  duties. 

Some  trainees,  but  not  all,  may  attend  additional  schools 
while  undergoing  FRS  training.  These  schools  are  attended  if  the 
periodic  requirements  for  completing  the  schools  occur  during  FRS 
training.  Among  these  schools  are  Survival,  Escape,  Resistance 
and  Evasion  (SERE)  School,  Flight  Instruments  Refresher  Course, 
and  Water  Survival  Training. 

The  combat  capable  training  courses  are  designed  to  produce 
fully  NATOPS  qualified  pilots  and  RIOs.  Trainees  are  evaluated  in 
oral  and  open  and  closed  book  exams,  in  normal  and  emergency 
procedures  using  the  COT  and  WST,  and  in  flight. 
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SECTION  II 

INSTRUCTIONAL  SYSTEM  DEVELOPMENT 


BACKGROUND 

Traditionally,  training  programs  within  the  military  have  been 
generated  in-house  by  a number  of  instructor  personnel  tasked  with 
the  difficult  problem  of  sorting  through  all  available  information 
and  developing  an  approach  to  the  classroom  presentation  of  an 
assigned  content  area.  The  resulting  instructor  guides  (outlines) 
and  student  handouts  are,  by  necessity,  based  on  information  within 
each  individual's  realm  of  experience.  This  experience  is  largely 
based  on  prior  instruction  and  specialized  operational  missions. 

Of  particular  significance  in  such  an  approach  to  training  is  a 
failure  to  conceptualize  the  entire  system  in  perspective.  This 
results  in  a piecemeal,  poorly-coordinated  buildup  of  instructional 
units  with  retrofitting  as  the  method  of  necessity  for  curriculum 
development.  Furthermore,  the  Subject  Matter  Experts  (SMEs) 
approaching  the  task  usually  do  not  have  the  benefit  of  expertise 
in  the  principles  and  application  of  instructional  technology. 

The  problem  is  complicated  by  the  rapid  turnover  in  instructor 
personnel.  Often  too,  an  attempt  is  made  to  piece  together  the 
in-house  training  materials,  typically  instructor  guides  (outlines), 
with  separately  developed  technical  materials  provided  by  a 
contractor  to  complement  existing  hardware.  The  classroom  instructor 
is  ultimately  left  with  the  task  of  integrating  the  curriculum 
while  performing  within  the  confines  of  the  existing  policy  and 
procedure  of  the  training  command. 

As  a result  of  this  piecemeal  procedure,  most  instructional 
programs  could  be  characterized  as  having  one  or  more  of  the 
following  problems: 

0 Critical  content  is  underemphasized  or  is  left  out. 

0 Instruction  and  testing  are  oriented  toward  general 
content  knowledge  rather  than  toward  specific  know- 
ledge related  to  operational  performance. 
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o 

o 

0 

0 

0 

ISD  evolved  in  response  to  these  symptoms,  as  well  as,  the 
increasing  demand  for  greater  efficiencies  in  a time  of  tightening 
resources.  It  is  an  attempt  to  derive  orderly  procedures  for 
the  design,  development,  implementation,  and  evaluation  of  in- 
structional systems,  and  to  provide  operational  guidelines  for 
carrying  out  the  procedures. 

Many  models  have  been  formulated  over  the  past  20  years. 

Most  have  contained  analysis,  design,  development,  implementation, 
and  quality  control  phases  tied  together  in  a sequential  flow. 

Some  have  been  more  prescriptive  than  others.  Some  have  failed; 
others  have  been  partially  successful;  but  not  one  has  received 
general  acceptance.  Although  the  progress  has  been  slow,  ISD 
technology  is  developing  as  data  on  the  various  methods  are 
collected. 

The  model  being  implemented  in  the  F-4  program  represents  a 
large  improvement  over  earlier  approaches.  It  contains  well- 
defined  procedures  for  some  tasks,  while  others  are  less  defined. 
There  are  many  areas  for  potential  improvement  in  the  model. 
Identifying  some  of  these  areas  is  one  of  the  goals  of  this  program. 

ASSUMPTIONS 

The  following  assumptions  underlie  the  ISD  model  which  was 
used  in  this  program. 


Learning  outcomes  and  evaluation  criteria  are 
poorly  defined. 

Instructional  methods  and  media  are  selected  with 
little  regard  for  relating  subject  matter  content 
to  media  capabilities. 

Subject  matter  content  is  instructor-specific 
and  changes  as  instructors  change. 

There  is  little,  if  any,  distinction  between 
content  and  instructional  design  experts,  i.e. 
content  experts  do  everything. 

Evaluation  and  update  procedures  are  rarely 
formulated. 
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0 The  resultant  training  courses  should  teach  knowledge 
and  skills  which  are  required  on  the  job. 

0 The  courses  when  implemented  should  be  efficient  in 
terms  of  resource  requirements. 

0 The  instructional  program  should  be  based  on  clearly- 
defined  objectives. 

0 Selection  of  instructional  methods  and  media  should 

be  based  on  a mapping  of  method  and  media  characteristics 
into  the  training  requirements  of  the  objectives. 

0 Solid  testing,  evaluation,  and  revision  procedures  are 
essential  to  success. 

Subject  matter  content  can  be  best  supplied  by 
personnel  experienced  in  system  operation.  To  insure 
optimal  use  of  their  expertise,  training  specialists 
should  channel  this  knowledge  of  the  system  into  the 
appropriate  ISD  formats. 


METHODS/PROCEDURES 

The  "6.2  Model"  is  the  state-of-the-art  in  ISD.  It  is  the 
most  precisely  defined  and  procedural ly  accurate  approach  to  ISD 
yet  developed.  The  detailed  specifications  for  the  model  follow 
a sequential  flow  beginning  with  task  listing  procedures  and 
terminating  with  instructor  training  for  implementation  (Figure  1). 
The  first  step  which  is  not  part  of  the  model,  is  the  problem 
analysis.  The  purpose  of  the  problem  analysis  is  to  provide 
background  information  on  the  strengths  and  weaknesses  of  the 
existing  training  system  (if  there  is  one).  Using  this  information 
it  can  be  determined  if  efforts  should  be  made  to  upgrade  the 
program  and,  if  so,  what  the  approach  should  be.  Not  all  problem 
analyses  identify  ISD  as  the  most  appropriate  and  cost-effective 
option.  Other  options  are  no  action  at  all,  updating  the  media 
used  without  doing  a full  analysis,  and  performing  an  ISD  analysis 
on  selected  portions  of  the  program.  In  addition,  the  problem 
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analysis  provides  background  information  concerning  training 
program  users  and  available  training  assets.  This  information 
assists  the  ISD  contractor  in  evaluating  the  total  environment 
in  which  the  upgraded  training  program  will  be  designed  and 
implemented. 

The  first  step  in  the  ISD  model  is  to  develop  a task  listing. 
This  step  helps  insure  that  all  tasks  that  should  be  included  in 
the  training  program  are  identified.  The  task  listing  is  used  in 
a job  analysis  survey  which  is  administered  to  personnel  who  are 
qualified  in  the  system  on  which  the  ISD  program  is  being  executed. 
During  the  job  analysis  survey,  these  personnel  are  questioned  on 
the  criticality  of  each  task  to  successful  performance,  the 
frequency  with  which  the  task  is  performed  on  the  job,  the 
difficulty  of  the  task,  where  the  task  was  learned  (i.e.,  in  a 
school  or  on  the  job)  and  where  the  task  should  be  taught.  The 
results  of  the  job  analysis  survey  are  used  to  determine  which 
tasks  should  be  included  in  the  training  program  and  the  emphasis 
which  should  be  placed  on  those  tasks  selected  for  training  (i.e., 
indepth  versus  refresher  training). 

Once  the  task  listing  has  been  validated  and  tasks  have  been 
selected  for  training,  the  tasks  are  used  as  the  basis  for 
developing  objective  hierarchies  and  Specific  Behavioral  Objectives 
(SBOs).  This  step  involves  analyzing  tasks  from  the  task  listing 
in  greater  detail  to  identify  all  skills  and  knowledge  which  are 
prerequisite  to  performing  the  tasks  identified  in  the  task  listing. 
The  structure  of  the  hierarchies  is  such  that  all  the  component 
objectives  of  a major  task  are  arranged  from  the  simplest  and 
most  fundamental  knowledge  up  to  the  most  complicated  skill.  Each 
entry  in  the  hierarchies  is  an  SBO  which  specifies  decisions  that 
must  be  made,  pieces  of  information  that  must  be  learned,  procedures 
that  must  be  executed,  or  skilled  actions  which  must  be  performed. 
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For  each  SBO,  conditions  under  which  the  objectives  must  be  per- 
formed and  the  minimum  acceptable  performance  standards  are 
specified.  By  laying  the  objectives  out  in  hierarchy  diagrams, 
the  subordinate,  superordinate,  and  coordinate  relationships 
among  the  objectives  can  be  clearly  identified.  The  objectives 
hierarchies  and  SBOs  provide  the  fundamental  data  base  on  which 
the  training  program  will  be  structured.  They  also  are  the  direct 
inputs  into  the  next  two  steps  in  the  !SD  process,  media  selection 
and  syllabus  development. 

Media  selection  involves  determining  which  of  the  candidate 
media  is  best  suited  for  providing  instruction  for  each  of  the 
SBOs  identified  in  the  previous  ISD  step.  The  output  of  the  media 
selection  process  is  a list  of  acceptable  instructional  media  for 
each  objective.  Acceptable  media  are  listed  in  order  of  preference. 

The  goal  of  this  process  is  to  select  media  which  will  enable  the 
best  possible  trainee  performance  while  minimizing  costs.  The 
procedures  to  be  employed  in  media  selection  have  yet  to  be 
standardized  in  the  6.2  model  and  are,  thus,  still  evolving.  In 
the  F-4  program  media  were  selected  on  the  basis  of  the  level 
of  behavior  expected  of  the  student,  the  level  of  content  being  taught, 
the  number  of  instances  the  student  should  see,  the  minimum  dis- 
play requirements,  and  the  amount  of  memorization  required.  Other, 
more  conventional,  media  selection  models  use  procedures  in  which 
the  display  and  fidelity  requirements  of  each  objective  are  mapped 
into  the  display  and  fidelity  characteristics  of  candidate  media. 

The  current  version  of  the  ISD  model,  MIL-T-29053,  does  not  specify 
a specific  procedure  to  be  used.  Rather  it  stipulates  that  an 
explicit  media  selection  model  shall  be  provided  for  approval  by 
the  contracting  agency. 

After  lists  of  acceptable  instructional  media  have  been 
specified  for  the  objectives,  the  objectives  are  grouped  and 
sequenced  to  form  the  syllabus.  Objectives  are  grouped  into  lessons 
and  the  lessons  are  sequenced  to  form  units,  which  contain  lessons 


24 


n 


NAVTRAEQUIPCEN  77-C-0081-1 

dealing  with  the  same  general  subject  matter  content  area.  Units 
are  sequenced  to  form  the  syllabus.  The  resultant  syllabus  should 
possess  three  characteristics.  First,  objectives  should  be 
sequenced  so  that  those  objectives  which  are  lower  in  a hierarchy 
are  taught  prior  to  higher  level  objectives  in  the  same  hierarchy, 
i.e.,  instruction  in  the  prerequisite  skills  and  knowledges  precedes 
instruction  in  the  more  complex  skills  or  decision  processes. 

Second,  whenever  possible,  the  sequence  should  specify  early  hands- 
on  learning  and  practice  and  maintain  an  integration  of  academic 
instruction  and  skills  practice  throughout  the  syllabus  sequence. 
Third,  in  general,  major  tasks  should  be  sequenced  from  easiest 
to  hardest.  When  feasible,  however,  those  tasks  which  will  reo'iire 
extensive  practice  should  be  introduced  early  in  the  syllabus  to 
allow  the  required  practice  to  be  integrated  throughout  the  remainder 
of  the  syllabus.  Applying  these  three  guidelines  during  lesson 
organization  and  sequencing  yields  a flow  of  instruction  in  which 
trainees  first  learn  operating  procedures,  then  practice  the 
procedures,  and  finally  perform  the  procedures  in  more  complex 
situations  in  which  decisions  and  integration  with  other  procedures 
are  required. 

At  this  point  in  the  sequence  of  ISD  steps  that  portion  of 
the  analysis  phase  which  involves  the  production  of  end  products 
which  will  be  converted  into  instructional  materials  has  been 
completed.  The  final  step  in  analysis  is  preparation  of  a Training 
Support  Requirements  Analysis  (TSRA).  The  TSRA  provides  an  estimate 
of  the  personnel,  equipment,  materials,  and  services  which  will 
be  required  during  subsequent  phases  in  the  ISD  program.  Separate 
estimates  for  design,  development,  implementation,  and  evaluation 
and  maintenance  are  provided.  The  TSRA  is  used  as  a planning 
document  for  budgeting  purposes. 

Due  to  the  detail  provided  on  the  number  of  lessons  per  in- 
structional medium  in  the  syllabus  and  the  cost  of  production  per 
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lesson  in  each  medium,  the  TSRA  can  be  used  to  identify  relative  media 
costs.  By  using  relative  costs,  areas  of  possible  cost  savings  can 
be  determined.  For  example,  the  production  of  slide-tape  materials 
is  relatively  costly  compared  to  workbooks  or  mediated  interactive 
lectures.  Workbooks  and  mediated  interactive  lectures  are  in  most 
cases  alternative  choices  for  slide-tapes.  If  there  are  costing 
constraints  for  production  and  revision  of  materials,  some  of  those 
lessons  for  which  tape-slide  was  first  chosen  may  instead  be 
presented  in  workbooks  or  mediated  interactive  lectures. 

Another  area  in  which  the  TSRA  provides  information  is 
facilities  requirements.  This  is  particularly  important  in  the 
planning  for  an  implementation  in  which  major  modifications  to 
existing  facilities  or  the  construction  of  new  facilities  will  be 
required.  In  order  to  have  the  facilities  available  when  all 
materials  have  been  developed  and  the  course  is  ready  for  imple- 
mentation, preparation  of  facilities  must  begin  as  early  as 
possible  after  the  initial  ISD  analyses  have  been  performed.  The 
TSRA  is  the  jumping  off  point  from  initial  analysis  into  the 
much  more  costly  and  lengthy  design,  development,  implementation, 
and  revision  phases. 

The  design  phase  involves  the  development  of  lesson 
specifications  for  all  lessons  in  the  syllabus.  A lesson 
specification  contains  the  subject  matter  content  and  outlines 
the  instructional  strategy  for  each  objective  in  the  lesson.  It 
is  during  this  step  that  the  information  that  will  be  presented 
in  the  course  is  assimilated  and  the  strategies  that  will  be  used 
in  the  presentation  of  the  lesson  and  testing  of  its  content  are 
determined.  In  addition  to  a statement  of  the  subject  matter 
content  of  each  objective,  a lesson  specification  contains  helps 
that  may  be  used  in  remembering  or  applying  the  objective,  the 
number  and  types  of  examples  required  to  support  instruction  of 
each  objective,  the  practice  and  testing  requirements,  and 
graphic  specifications. 
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After  the  design  phase  is  completed,  the  development  phase 
begins.  During  the  development  phase  a number  of  tasks  are 
carried  on  concurrently.  The  primary  task  involves  converting 
the  lesson  specifications  into  lesson  and  testing  materials. 

Authors  use  the  lesson  specifications  in  conjunction  with  lesson 
guides  which  explain  how  to  produce  instructional  material  for 
each  medium  being  used.  Lesson  materials  are  produced  in  draft 
form  and  reviewed  by  the  instructional  psychologist.  Prototype 
materials  are  then  produced  for  small  scale  tryout.  Based  on  data 
collected  during  the  small  scale  tryout,  lesson  materials  are 
revised  for  final  production. 

While  the  instructional  materials  and  tests  are  being 
developed,  quality  control  and  implementation  plans  must  be  form- 
ulated. These  plans  help  insure  that  implementation,  evaluation, 
and  revision  procedures  are  wel 1 -formulated  and  are  workable  within 
the  training  environment.  The  quality  control  plan  presents  the 
procedures  to  be  followed  in  continuously  assessing  the  relevance, 
effectiveness,  and  currency  of  the  instructional  materials  and 
the  instructional  management  system.  It  provides  for  such 
assessment  from  the  earliest  materials  development  stage  to  the 
implementation  of  the  training  program  and  throughout  the  life 
cycle  of  the  weapons  system.  The  quality  control  plan  addresses 
three  major  areas:  formative  revision,  internal  quality  control 
(trying  out  the  finished  materials  with  real  students  in  the  real 
training  environment),  and  external  quality  control  (the  validity 
of  the  training  program  in  terms  of  its  relevance  to  operational 
requirements  for  skilled  personnel). 

The  implementation  plan  details  the  procedures  and  milestones 
which  will  be  followed  during  the  implementation  phase  of  the 
ISD  program.  The  plan  focuses  on  training  management  and 
administration,  personnel  requirements,  and  implementation  procedures. 
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Some  of  the  specific  content  areas  contained  in  the  implementation 
plan  are: 

0 Instructional  system  integration 
0 Equipment  storage,  maintenance,  and  utilization 
0 Student  grading  evaluation 
0 Training  materials  management 
0 Facilities  utilization 
0 Student  scheduling  and  management 
0 Schedule  contingency  plan 

The  final  step  during  the  development  phase  is  the  preparation 
of  an  Instructor  Training  Course  (ITC).  The  ITC  is  designed  to 
teach  Fleet  Readiness  Squadron  (FRS)  personnel  to  properly  use 
the  instructional  system.  It  provides  instruction  in  the  following 
areas: 

0 The  training  organization  to  include  personnel, 
organization,  policies,  and  student  characteristics 

0 The  training  program  to  include  syllabus  organization 
and  student  flow,  instructor  duties,  media  used  in 
the  training  program,  student  learning  center  operations, 
ISD  team  duties,  and  an  overview  of  the  ISD  process. 

0 Training  system  operation  and  scheduling 

0 Storage,  operation,  and  troubleshooting  nf  media 
devices. 

0 Conducting  lessons  in  all  media  used  in  the  program 
0 Evaluation  of  students  and  instructional  materials 
0 Revision  of  instructional  materials 

After  all  instructional  materials  have  been  produced,  the 
quality  control  and  implementation  plans  have  been  formulated, 
and  the  ITC  has  been  prepared  and  administered,  the  implementation 
phase  begins.  The  length  of  implementation  is  a function  of 
course  duration  and  the  Interval  between  class  convenings.  At  a 
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quality  control  data  is  collected  on  that  class.  Preferrably 
its  duration  exceeds  that  of  one  class  so  that  data  on  multiple 
classes  can  be  collected  and  necessary  modifications  to  the 
training  materials  and/or  the  management  plan  can  be  made.  Most 
of  the  "bugs"  should  be  worked  out  of  the  new  training  system 
during  this  initial  implementation  phase. 

The  final  phase  in  the  ISO  process  is  evaluation  and  main- 
tenance of  the  training  program  for  as  long  as  it  is  in  use. 

While  this  phase  is  less  formal  than  the  previous  phases  in  the 
concentration  on  ISO  methodology,  it  embodies  a continuation  of 
ISD  procedures  in  the  constantly  ongoing  process  of  updating  the 
training  program.  This  is  a particularly  crucial  phase  since  +he 
structure  of  the  program  will  change  and  possibly  erode  if  a nigh 
quality  maintenance  program  is  not  provided.  The  procedures  de- 
tailed in  the  quality  control  and  implementation  plans  and  the 
instructor  training  course  must  be  maintained.  This  function  may 
be  allocated  to  a small  ISD  team  which  should  be  well -schooled 
in  ISD  procedures  in  general  and  in  the  specifics  of  the 
particular  training  program. 

The  scope  of  this  F-4  ISD  program  encompasses  the  analysis 
phase  of  the  ISD  process.  The  task  listing  was  generated  by 
MCCTRG-10  ISD  personnel  under  the  guidance  of  Courseware,  Inc. 

It  provided  the  primary  input  into  hierarchy  and  SBO  development, 
which  was  the  first  task  of  this  contract.  Subsequent  tasks  were 
media  selection,  syllabus  development,  and  training  support 
requirements  analysis.  How  these  tasks  were  carried  out  is 
discussed  in  Section  III,  Implementation. 
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SECTION  III 
IMPLEMENTATION 


ORGANIZATION  AND  STAFFING 

PERSONNEL.  Three  organizations  participated  in  the  F-4  ISO 
program.  These  organizations  and  their  roles  are  as  follows: 


0 Contractor 


0 MCCTRG-10  ISD  Dept, 
(supplemented  by 
VF- 121 ) 


-On-site  training,  guidelines, 
and  review  of  products  in 
accordance  with  ISD  procedures 

-Supply  subject  matter  experts 
to  provide  input  on  F-4  procedures, 
tactics,  etc.  Provide  facilities 
for  the  project. 


0 NAVTRAEQUIPCEN  -Project  coordination,  direction 

and  monitoring 


Under  provisions  of  the  contract  MCCTRG-10  was  tasked  to 
provide  Marine  subject  matter  experts  (SMEs)  to  the  program.  Four 
SMEs  participated  on  a part-time  basis  for  the  first  26  weeks  of 
the  32-week  program.  SMEs  were  not  required  for  the  final  six 
weeks.  In  addition,  three  Navy  SMEs  from  VF-121,  NAS  Miramar, 
participated  on  a non-overlapping  basis  for  a total  of  ten  weeks. 
On-site  contractor  personnel  included  an  instructional  psychologist, 
who  was  the  on-site  supervisor,  a former  F-4  RIO,  who  assisted 
the  on-site  supervisor  in  coordinating  all  project  activities  and 
who  reviewed  outputs  from  the  Marine  and  Navy  SMEs,  and  a secretary. 
The  program  director  was  located  off-site.  He  made  monthly  visits 
to  MCAS,  Yuma,  reviewed  all  products  of  the  program,  and  was 
responsible  for  liaison  with  NAVTRAEQUIPCEN.  Overall  responsibility 
for  the  program  rested  with  the  Allen  Corporation  Technical 
Director.  He  reviewed  all  products  of  the  program.  The  hierarchical 
organization  of  the  program  team  is  shown  in  Figure  2. 
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As  indicated  by  the  dashed  lines  in  Figure  2,  the  activities 
of  the  Marine  and  Navy  SMEs  were  governed  by  the  existing  military 
command  structure.  They  reported  to  the  head  of  the  MCCTRG-10 
ISD  Department,  who  was  responsible  for  coordination  with  on-site 
contractor  personnel . 

PROJECT  SEQUENCING.  The  sequencing  of  major  project  activities 
was  planned  at  the  outset  as  shown  in  Figure  3.  The  project  was 
contractually  scheduled  to  be  a 32-week  effort.  Hierarchy  and 
SBO  development  was  scheduled  for  11  weeks.  Media  selection, 
syllabus  development,  and  the  TSRA  were  scheduled  for  four  weeks 
each.  The  right  sides  of  the  hierarchy  and  SBO  development, 
media  selection,  syllabus  development,  and  TSRA  blocks  indicate 
the  submission  date  for  a draft  report.  In  each  case  the  final 
report  for  that  activity  was  scheduled  four  weeks  after  draft 
report  delivery. 

PROJECT  ACTIVITIES 

INTRODUCTION.  In  the  following  sections  detailed  descriptions  of 
each  of  the  major  ISD  activities  conducted  during  the  project 
will  be  discussed.  For  each  activity  the  following  categories 
of  information  are  included: 

0 An  overview  of  the  purpose  of  the  activity 
0 A description  of  the  procedures  employed 
0 A description  of  SME  training 
0 An  account  of  the  problems  and  solutions 
The  last  section  contains  a summary  of  the  F-4  training 
courses  developed  during  this  program. 

OBJECTIVES  HIERARCHIES  AND  SBOs. 

Overview.  The  first  major  task  performed  under  this  contract 
was  the  development  of  objectives  hierarchies  and  SBOs.  A 
hierarchy  is  a visual  layout  of  the  structure  of  the  component 
objectives  which  support  the  performance  of  a task.  It  shows 
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subordinate,  superordinate,  and  coordinate  relationships  among 
the  objectives.  The  procedures  used  to  develop  hierarchies  of 
objectives  and  tasks  provide  a systematic  way  of  determining 
what  the  student  must  learn  in  order  to  perform  his  job.  They 
constitute  the  data  base  which  will  be  used  in  the  further 
development  of  the  training  program.  A training  program  developed 
from  objectives  hierarchies  is  more  likely  to  be  complete  in  its 
job  and  task  orientation  and  free  of  irrelevant  information,  which 
is  time  consuming  to  teach  but  adds  little  to  the  student’s 
ability  to  perform  the  job. 

Procedures.  The  input  to  the  hierarchy  and  SBO  development  process 
is  the  validated  task  listing.  It  contains  the  major  tasks  which 
have  been  selected  for  training  during  job  analysis.  For  this 
program  the  pilot  and  RIO  task  listings  were  furnished  by  the 
government.  They  were  structured  by  missions  and  segments  within 
missions.  There  were  four  missions  with  ten  mission  segments  per 
mission.  A visual  representation  of  the  mission  by  segment 
structure  of  the  task  listing  is  shown  in  Figure  4.  The  entries 
in  the  cells  are  the  mission  by  mission  segment  numbers.  These 
numbers  formed  the  basis  for  the  numbering  system  which  was  used 
during  hierarchy  development. 

The  procedures  used  to  develop  the  objective  hierarchies  are 
outlined  in  Figure  5.  An  elaboration  of  the  steps  shown  in  Figure 
5 follows: 

Step  1:  List  major  tasks  - This  step  was  accomplished  during 
task  listing.  It  is  included  here  to  emphasize  that  the  task  is 
the  input  to  the  objective  hierarchy  and  SBO  development  process. 

Step  2:  Select  the  most,  or  next  most  complicated  task 
for  analysis  - The  following  criteria  should  be  used  in  the  selection 
of  the  most  complicated  task: 

0 Select  a task  which  involves  a large  part  of  the  job 
time 

0 Select  a task  which  requires  many  sub-procedures  for 
its  completion 

.15 


SEGMENT 


Figure  5.  The  Process  of  Objectives 
Hierarchy  Development 
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0 Select  a task  which  tends  to  be  one  of  the  most 
difficult 

Step  3:  List  all  components  of  the  major  task  - Analyze 
the  performance  components  of  the  selected  major  task  and  divide 
the  major  task  into  its  component  subtasks.  This  will  yield  the 
steps  or  procedures  required  to  perform  the  major  task.  Steps 
or  procedures  may  be  decisions,  as  well  as,  overt,  observable  acts. 

Step  4:  Determine  whether  all  necessary  decisions  have  been 
identified  as  task  components  - When  analyzing  a task  that  requires 
performance  of  a large  number  of  procedures  and  interaction  with 
complex  system  hardware,  identification  of  each  operator  decision 
is  made  to  determine  whether  the  operator  must  decide  when  to 
perform  a procedure,  decide  which  of  multiple  rules  or  procedures 
to  use,  evaluate  the  adequacy  of  a procedure,  and/or  decide  when 
a procedure  should  be  stopped.  In  analyzing  a major  task,  decision 
subtasks  can  easily  be  omitted.  This  step  is  included  as  a cross- 
check of  Step  3 to  insure  that  all  decision  tasks  have  been 
included. 

Step  5:  Add  decision  components  to  the  list  - If  additional 
decision  subtasks  are  identified  in  Step  4,  they  should  be  added 
to  the  component  subtask  list  for  the  major  task.  When  incor- 
porated into  the  hierarchy  structure,  these  decision  subtasks  will 
be  on  the  same  level  as  the  performance  subtasks. 

Step  6:  Determine  whether  there  is  a significant  memorization 
requirement  - the  memorization  component  is  significant  if  an 
average  trainee  would  be  unable  to  perform  the  task  as  a whole 
because  he  could  not  remember  which  components  must  be  performed 
or  the  order  of  their  performance.  If  the  memorization  component 
for  the  task  is  significant,  it  will  be  incorporated  in  Step  7. 

Step  7:  Add  a component  requiring  recall  of  the  steps  of 
the  major  task  - If  there  is  a significant  memorization  component 
identified  in  Step  6,  it  should  be  added  to  the  component  subtask 
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totality  of  information  required  to  perform  the  task. 

Step  11:  Expand  the  set  of  components  - If  the  component 
subtask  listing  contains  fewer  than  the  minimum  number  of  sub- 
tasks required  to  perform  the  task,  it  should  be  expanded  to 
insure  that  the  completed  training  program  will  contain  all  in- 
formation necessary  to  train  students  to  perform  the  task. 

Step  12:  Write  an  instructional  objective  for  each  component 
An  instructional  objective  contains  three  elements,  the  subtask 
component  statement,  which  is  the  "action"  portion  of  the  objective 
the  conditions,  and  the  standards.  Conditions  describe  the 
circumstances  under  which  the  subtask  will  be  performed.  Types  of 
conditions  statements  are  system  configuration,  environmental 
factors,  equipment  factors,  relevant  available  information,  and 
initiation  stimulus.  Standards  are  the  criteria  for  minimum 
acceptable  subtask  performance.  Types  of  standards  statements 
are  time,  accuracy/error  rate,  degree  of  conformity  with  specified 
procedures,  and  safety  considerations. 

Step  13:  Arrange  objectives  in  the  hierarchy  - Objectives 
are  arranged  under  the  superordinate  objective  that  is  being 
analyzed.  All  are  one  level  below  the  superordinate  objective. 

Step  14:  Determine  whether  each  objective  is  a basic  level 
behavior  - An  objective  is  a basic  level  behavior  if  it  is  the 
lowest  level  behavior  that  is  significant  for  training.  If  all 
objectives  are  basic  level  behaviors,  the  next  complicated  major 
task  is  selected  from  the  task  listing  and  the  process  is  repeated. 

Step  15:  Select  one  basic  level  subtask  - If  one  or  more 
objectives  are  not  basic  level  behaviors,  one  objective  is  chosen 
and  subtask  components  for  that  objective  are  listed.  This 
process  is  iterated  until  all  of  the  lowest  level  objectives  have 
been  identified  for  the  major  task. 

Examples  of  a hierarchy  diagram  and  an  SBO  page  are  shown  in 
Figures  6 and  7 respectively.  A detailed  description  of  the 
numbering  and  coding  conventions  used  in  the  hierarchies  and 
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MISSION:  AIR  TO  AIR 

SEGMENT:  CONDUCT  PREMISSION  PLANNING  PHASE 
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SBOs  plus  additional  examples  are  presented  in  Appendix  A. 

SME  Training.  Since  a previous  contractor.  Courseware,  Inc., 
had  provided  SME  training  on  the  hierarchy  and  SBO  development 
process  prior  to  contract  award  to  Allen  Corporation,  minimum 
additional  SME  training  was  required.  A one-day  session  was 
conducted  to  insure  that  the  procedures  being  used  were  in  accord 
with  the  F-4  ISO  specification  and  that  the  quality  of  the  end- 
products  was  in  accord  with  Allen  Corporation  expectations.  The 
transition  between  contractors  went  smoothly,  and  there  were  no 
substantial  differences  in  approach  between  Courseware,  Inc.  and 
Allen  Corporation. 

Problems  and  Solutions.  The  hierarchy  and  SBO  development  process 
went  smoothly  with  only  a few  minor  difficulties.  One  of  these 
difficulties  involved  determining  the  lowest  levels  in  the 
hierarchies,  i.e.,  establishing  that  point  beyond  which  another 
level  of  objectives  would  be  trivial  and  above  which  the  hierarchies 
would  be  incomplete  if  terminated  at  that  point.  Although  the 
process  of  objectives  hierarchy  development  shown  in  Figure  5 does 
delineate  steps  (8,  9,  10,  and  11)  that  are  included  to  assist 
in  determining  the  appropriate  lowest  level  in  the  hierarchies, 
consistently  choosing  the  appropriate  level  is  difficult.  In 
order  to  help  remedy  this  difficulty,  SMEs  were  instructed  to 
evaluate  at  what  level  in  each  hierarchy  the  information  was 
complete  so  that  instruction  based  on  the  hierarchy  structure 
would  not  present  trivial  detail,  or  leave  out  instruction  which 
was  essential  for  performance  of  the  terminal  task.  This  was  done 
by  having  the  SMEs  think  through  the  training  that  would  be 
presented  to  support  each  enabling  objective  in  the  hierarchies. 

If  the  objective  did  not  clearly  "suggest"  all  training  content 
required,  more  objectives  were  added.  If  on  the  other  hand, 
trivial,  entry-level  objectives  were  contained  in  the  hierarchies. 
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they  were  deleted.  This  approach  also  assisted  the  SMEs  in 
understanding  the  relationship  of  the  hierarchies  and  SBOs  to 
the  development  of  training  materials. 

The  re-evaluation  of  the  completeness  and  accuracy  of  the 
task  listing,  which  was  a natural  by-product  of  hierarchy  and 
SBO  development,  led  to  numerous  changes  in  the  task  listing. 

This  is  an  illustration  of  the  well-known  fact  that  ISD  is  a 
constantly  iterative  process,  that  in  many  cases  prior  end 
products  require  changes  as  a result  of  further  analyses.  Such 
changes  are  particularly  prevalent  during  the  earlier  steps  in 
the  ISD  analysis  phase,  during  which  the  data  base  for  training 
course  development  is  being  established.  The  changes  were  re- 
corded in  a master  task  listing  document. 

The  conditions  and  standards  produced  by  the  SMEs  were  in 
many  cases  very  general  and  added  little  to  the  SBOs.  For  example, 
a frequently  used  condition  was  "Given  an  exam"  and  frequently 
used  standards  were  “To  criterion"  and  "In  accordance  with  NATOPS" 
(IAW  NATOPS).  With  additional  work  by  the  SMEs,  the  conditions 
and  standards  could  have  been  upgraded. 

The  more  fundamental  consideration,  however,  relates  to  the 
utility  of  specifying  conditions  and  standards  at  this  early  point 
in  the  analysis  phase.  The  primary  reason  for  including  conditions 
and  standards  is  to  insure  that  the  corresponding  objective  is 
observable  and  testable.  Given  that  the  objective  is  a valid 
prerequisite  to  performance  of  a terminal  objective,  there  will 
be  conditions  under  which  it  can  be  evaluated.  Also,  conditions 
and  standards  are  partially  a function  of  the  instructional  medium 
in  which  the  objective  will  be  learned.  In  the  ISD  process, 
media  selection  follows  hierarchy  and  SBO  development.  A precise 
determination  of  conditions  and  standards  takes  place  during  the 
development  of  segment  specifications  for  the  objectives.  At  this 
point,  media  have  been  selected  and  testing  philosophies  have  been 
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formulated. 

The  determination  of  conditions  and  standards  during 
hierarchy  and  SBO  development  detracts  from  the  primary  purpose 
of  structuring  hierarchies  which  provide  the  framework  for  the 
development  of  the  syllabus  and  the  ultimate  training  program. 

It  is  recommended  that  consideration  be  given  to  deleting  conditions 
and  standards  from  SBOs  written  in  conjunction  with  hierarchy 
development  and  emphasizing  the  importance  of  including  complete, 
accurate  conditions  and  standards  in  the  segment  specifications. 

MEDIA  SELECTION. 

Overview.  The  next  step  in  the  F-4  ISD  program  was  to  select 
appropriate  instructional  media  for  each  SBO  produced  during  the 
preceding  hierarchy  and  SBO  development  step.  This  step  provides 
a matching  of  the  instructional  capabilities  of  various  media  with 
the  characteristics  of  the  objective  as  they  relate  to  presentation 
and  learning  strategies.  It  is  important  that  this  mapping  of 
objective  requirements  onto  media  capabilities  be  as  accurate  as 
possible  within  cost  and  feasibility  constraints.  This  helps 
insure  that  trainees  are  presented  information,  afforded  practice, 
etc.,  in  an  optimal  learning  environment. 

The  algorithm  used  to  select  media  differed  from  more 
conventional  media  selection  procedures  in  that  level  of  expected 
behavior,  level  of  content,  amount  of  practice  required,  and  the 
amount  of  memorization  required  were  explicit  criteria  for 
selection.  In  addition,  presentation  display  requirements,  the 
nucleus  of  the  more  conventional  media  selection  models,  was 
included  as  one  of  the  components  of  the  algorithm. 

Procedures.  The  input  to  the  media  selection  process  was  the 
listing  of  all  SBOs.  Each  SBO  was  carried  through  the  selection 
algorithm  to  determine  the  primary  and  alternate  media  which 
should  be  used.  Alternate  media  were  provided  for  each  objective 
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to  allow  flexibility  during  later  phases  of  the  ISD  process  when 
modifications  based  on  cost  factors  or  syllabus  organization  may 
be  required. 

The  media  selection  model  used  in  this  program  involved 
asking  a series  of  five  questions,  the  answers  to  which  traced  a 
path  through  a network  to  a terminal  block  which  contained  a 
code  number  for  the  media  selected.  The  network  is  shown  in 
Tiguro  8 and  the  questions  used  to  determine  paths  through  the 
network  are  shown  in  figure  9.  The  question  blocks  in  the  network 
are  labelled  Ql,  1)2,  QJ,  Q4,  and  Q5. 

The  selection  process  begins  at  the  triangle  in  the  center 
of  the  network  and  goes  to  question  one  (Ql).  The  numbered 
response  to  Ql  determines  the  path  that  will  be  followed  to  the 
next  question.  In  the  same  manner,  at  each  of  the  other  four 
questions  (Q2  through  Q5)  the  procedure  continues  along  the  path 
associated  with  the  appropriate  answer  to  each  question. 

Each  path  leads  to  recommended  media,  which  are  the  terminal 
blocks  Ml  through  M44  on  the  network.  The  media  selections 
corresponding  to  the  numbers  in  the  terminal  blocks  are  presented 
and  defined  in  Appendix  B.  In  addition.  Appendix  B contains 
examples  of  completed  media  selection  diagrams. 

The  media  selection  process  was  conducted  primarily  by  the 
project  instructional  psychologist.  SMEs,  however,  provided 
significant  input  particularly  relating  to  the  difficulty  of 
the  objectives.  Questions  2 (amount  of  presentation  or  practice) 
and  5 (memorization  requirements),  in  essence,  assess  the 
difficulty  of  the  objective.  SMEs  were  also  called  upon  to 
answer  questions  relevant  to  objective  content. 

SHE  Training.  Since  media  selection  was  primarily  a contractor 
task,  no  extensive  SME  training  was  required.  An  informal  four- 
hour  course  was  conducted  to  acquaint  the  SMEs  with  the  selection 
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QUESTION  1: 

What  is  the  level  of  behavior  expected  of  the  student 
in  this  segment? 

1 = familiarization 

2 = discriminated  recall 

3 = rule-using 

QUESTION  2: 

What  level  of  content  is  being  taught  in  this  segment? 

1 = familiarization 

2 = paired  associate 

3 - concept 

4 = rule 

QUESTION  3: 

Is  the  minimum  critical  set  of  instances  the  student 
needs  to  see  small  or  large? 

1 = small 

2 = large 

QUESTION  4: 

What  is  the  minimum  display  requirement? 

1 = verbal  and/or  symbolic  and/or  static  simple 

pictorial 

2 = verbal  and/or  symbolic  and/or  static  complex 

pictorial 

3 = dynamic  pictorial 

4 = interactive 

QUESTION  5: 

Is  the  memorization  component  of  this  objective  large 
or  small? 

1 = small 

2 = large 


Figure  9.  Questions  and  Answers  Used 
in  Conjunction  with  the 
Network  Shown  in  Figure  8 
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process  and  to  relate  media  selection  to  other  steps  in  the 
ISD  process.  It  was  necessary  that  the  SMEs  understand  the 
procedures  in  the  model  so  that  they  could  answer  questions 
which  required  subject  matter  expertise. 

Problems  and  Solutions.  The  primary  problem  in  applying  the 
media  selection  model  was  the  ambiguity  of  some  of  the  responses 
to  the  five  questions.  Without  more  precise,  working  definitions 
than  those  supplied  in  the  F-4  ISD  specification  the  model  could 
not  be  applied  with  consistency  or  with  the  accuracy  required. 

In  order  to  remedy  this  problem,  Allen  Corporation  ISD  personnel 
consulted  with  the  Courseware,  Inc.  personnel  who  formulated 
the  selection  model.  The  resultant  definitions  of  the  terms 
made  the  model  workable;  however,  it  was  still  felt  that  further 
clarification  was  needed.  The  five  questions  and  elaborations  of 
the  possible  responses  to  each  are  as  follows: 

Question  1:  What  is  the  level  of  behavior  expected  of  the 
student  in  this  segment? 

1.  familiarization:  the  student  will  be  exposed  to  the 
information  or  skill,  but  need  not  be  intimately 
familiar  with  it  and  will  not  be  tested. 

2.  discriminated  recall:  the  student  will  remember,  recall, 
or  recognize  something  based  on  previously  presented 
information,  e.g.,  recall  operational  fuel  flow  rate. 

3.  rule  using:  the  student  will  apply  a rule  to  a given 
set  of  inputs  to  formulate  an  output,  e.g.,  dis- 
criminate among  types  of  threat  aircraft. 

Question  2:  What  level  of  content  is  being  taught  in  this  segment? 

1.  familiarization:  the  content  level  is  for  basic 
acquaintance  and  recognition  but  will  not  be  a 
test  item. 

2.  paired  associate:  this  content  level  involves  the 
teaching  of  a one-to-one  association  (presence  of 
A indicates  B is  occurring)  and  would  require 
future  recall  of  the  content. 
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3.  concept:  the  content  level  would  show  grouped  or  sets 

of  symbols  or  events  with  common  critical  characteristics 
or  names,  e.g.,  identify  an  engine  fire. 

4.  rule:  this  level  of  content  requires  the  manipulation 
of  a concept,  e.g.,  calculate  refueling  requirements. 

Question  3:  Is  the  minimum  critical  set  of  instances  the  student 

needs  to  see  small  or  large? 

Essentially  the  question  was  interpreted  to  say,  "How  often 
do  we  have  to  expose  the  student  to  this  content  during  his 
training  cycle  in  order  for  him  to  master  the  content  or 
how  often  must  a given  skill  be  practiced  to  insure  mastery?" 
Question  4:  What  is  the  minimum  display  requirement?  That  is, 
how  sophisticated  must  the  presentation  to  the  trainee  be  in  order 
to  adequately  convey  the  desired  content? 

1.  verbal  and/or  symbolic  and/or  static  simple  pictorial, 
e.g.,  a simple  photograph,  typed  or  recorded  statement, 
or  drawing. 

2.  verbal  and/or  symbolic  and/or  static  complex  pictorial, 
e.g.,  a complex  photograph,  typed  or  recorded  statement 
or  drawing. 

3.  dynamic  pictorial,  e.g.,  a moving  picture,  animated 
drawing,  computer  generated  image,  etc. 

4.  interactive:  Must  the  presentation  media  sense  student 
responses  and  respond  appropriately? 

Question  5:  Is  the  memorization  component  of  this  objective  large 

or  small? 

This  question  was  interpreted  as  a discrimination  between 
"small"  and  "large"  based  upon  content  complexity  and  volume. 
The  more  complex  objectives  involve  certain  detailed 
memorization  factors,  e.g.,  navigation  would  contain 
components  requiring  "large"  memorization.  Other  objectives 
that  are  assisted  by  supportive  documentation  such  as 
checklists,  were  categorized  as  containing  a "small" 
memorization  component. 
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If  this  media  selection  model  is  to  be  used  in  future  ISD 
programs,  it  is  recommended  that  more  precise  operational  definitions 
be  used  in  place  of  those  currently  supplied.  Incorrect  inter- 
pretations of  the  five  questions  and  their  sets  of  responses  can 
have  significant  effects  on  the  media  selected.  This  could  lead 
to  teaching  objectives  in  media  which  are  not  optimally  suited  to 
the  learning  requirements  of  the  objectives. 

The  possible  set  of  media  which  could  be  selected  for  use 
in  the  F-4  training  program  was  constrainted  to  the  candidate  media 
contained  in  the  model.  This  constraint  led  to  a "closed"  selection 
process  rather  than  an  "open"  process  in  which  a much  larger  set  of 
candidate  media  could  have  been  considered.  For  example,  among 
other  feasible  alternatives  for  this  program  are  audio  tapes  with 
workbooks  and  continuous  loop  8 mm.  systems,  which  can  provide 
either  frame-by-frame  projection  or  continuous  motion. 

The  area  of  media  selection  is  a difficult  one  at  best. 

There  are  almost  as  many  selection  models  as  there  are  media  types. 

As  yet,  there  is  no  completely  satisfactory  approach  to  this 
problem.  This  contractor  has  not  developed  an  alternative  approach 
to  media  selection;  however,  within  the  context  of  the  F-4  model 
several  suggestions  can  be  made.  Rather  than  selecting  specific 
media  it  is  recommended  that  a class  of  media  be  selected,  all 
members  of  which  have  similar  presentation  capabilities.  Once 
a class  of  media  is  selected  for  an  objective,  trade-off  studies 
can  be  performed  to  determine  which  medium  in  the  class  is  best 
suited  in  terms  of  specific  capabilities  and  cost  factors.  Cost 
is  not  considered  in  the  present  model  and  should  be  evaluated 
during  the  media  selection  process.  For  example,  data  in  the 
Training  Support  Requirements  Analysis  indicate  that  the  relative 
cost  of  slide  tapes  and  workbooks  are  about  four  to  one.  Yet 
slide-tapes  and  workbooks  are  similar  in  instructional  capabilities. 
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In  the  F-4  ISO  specification,  media  selection  and  syllabus 
development  are  discrete,  sequential  processes  with  media  selection 
preceding  syllabus  development.  They  are,  thus,  two  independent 
processes  with  the  results  of  media  selection  having  minimum  impact 
on  lesson  organization  and  sequencing.  The  structuring  of  lessons 
is  done  on  the  basis  of  subject  matter  content  rather  than  on  the 
basis  of  media  selected  for  individual  objectives.  Under  the 
current  specification  lessons  are  organized  by  content.  The  media 
for  the  objectives  in  the  lesson  are  then  examined  to  determine 
if  more  than  one  medium  was  selected  for  the  grouped  objectives. 

•If  so,  the  predominant  medium  from  those  assigned  to  the  objectives 
in  the  lesson  is  selected.  This  procedure  is  inefficient  since 
some  of  the  media  selected  for  individual  objectives  are  changed 
to  be  consistent  with  the  lesson  content  requirements.  Media 
selected  for  objectives,  grouped  into  a lesson  on  the  basis  of 
content  and  context  of  training,  should  fall  within  the  same 
class.  If  this  does  not  happen,  it  is  indicative  that  the  lesson 
has  been  poorly  organized. 

It  is  recommended  that  media  be  selected  for  lessons  rather 
than  for  individual  objectives,  thus  reversing  the  order  of  media 
selection  and  syllabus  development.  Two  advantages  of  this  approach 
are:  (1)  the  process  has  to  be  carried  out  fewer  times  since  there 
are  fewer  lessons  than  there  are  objectives  and  (2)  once  made, 
media  selections  will  be  final  rather  than  subject  to  change 
during  a subsequent  task  in  the  ISO  process. 

SYLLABUS. 

Overview.  The  development  of  pilot  and  RIO  syllabi  was  the  last 
step  in  the  analysis  phase  which  provided  input  directly  into 
lesson  specification  development  (Figure  1).  The  other  remaining 
analysis  step  was  the  Training  Support  Requirements  Analysis 
(TSRA)  which  provided  personnel,  materials,  and  services  cost 
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estimates  for  the  four  remaining  phases,  but  which  was  not  directly 
related  to  the  subsequent  phases  in  terms  of  technical  steps 
specified  for  course  development. 

Syllabus  development  was  the  process  of  grouping  the  SBOs 
from  the  hierarchy  and  SBO  development  step  into  lessons,  and 
sequencing  the  lessons  to  form  the  pilot  and  RIO  syllabi.  The 
resultant  syllabi  formed  the  basis  for  continued  development  of 
the  two  training  courses.  Each  lesson  constituted  a discrete 
block  of  instruction  with  the  objectives  contained  in  the  lesson 
providing  an  outline  of  the  subject  matter  content  to  be  covered 
in  ground  school  or  the  skills  to  be  practiced  in  simulator  and 
flight  lessons.  The  goal  of  syllabus  development  was  to  arrange 
all  objectives  so  as  to  facilitate  administration,  scheduling, 
sequencing,  testing,  and  learning. 

Procedures.  Each  course  was  structured  into  three  levels  of 
organization.  These  levels  ordered  from  smallest  to  largest  are 
as  follows: 

0 Segment  - The  instruction  required  to  teach  one  objective. 

A segment  is  comparable  to  an  objective  and  is 
worded  the  same  as  an  objective.  During  syllabus 
development  the  terminology  is  changed  from 
"objective"  to  "segment"  to  indicate  that  each 
objective  is  now  thought  of  as  a segment  of 
instruction.  During  lesson  specification 
each  segment  will  be  the  starting  point  for 
detennining  the  subject  matter  content  and 
instructional  strategy  which  will  be  used  to 
teach  the  objective. 

0 Lesson  - A group  of  related  segments  with  a required 
testing  point.  The  F-4  ISD  specification 
contains  guidelines  for  the  number  of  segments 
which  should  constitute  a lesson.  The  range 
is  from  3 to  15  objectives.  More  importantly, 
a lesson  snould  be  composed  of  segments  which 
form  a logical  grouping  in  terms  of  content 
and  can  be  taught  at  one  sitting. 
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0 Unit  - A group  of  related  lessons  with  a required 
testing  point.  This  is  the  largest  sub- 
division within  the  syllabus.  Units  contain 
lessons  grouped  on  the  basis  of  subject  matter 
content  and  ordered  so  that  earlier  lessons  are 
prerequisite  to  latter  lessons  in  the  same  unit. 

The  structure  of  units  thus  has  a large  impact 
on  the  effectiveness  of  training.  Students 
should  have  mastered  prerequisite  skills  and 
knowledge  before  being  exposed  to  more  compli- 
cated learning  situations.  For  example, 
maneuvers  to  be  performed  in  the  aircraft 
should  first  be  discussed  in  a ground  school 
lesson  and  practiced  in  a simulator  (if  simulator 
capabilities  permit)  before  being  practiced  in 
the  aircraft.  By  using  this  sequence  the  trainee 
is  better  prepared  to  make  efficient  use  of  the 
most  expensive  training  device,  the  aircraft. 
Finally,  in  sequencing  units  within  the  syllabus, 
the  same  guidelines  as  those  used  in  sequencing 
lessons  within  units  should  be  used,  i.e.,  establish 
prerequisites  by  units.  For  example,  a unit  on 
basic  aircraft  maneuvering  should  precede  a unit 
on  air-to-air  combat. 

The  total  syllabus  development  process  has  two  components, 
organization  of  segments  into  lessons  and  sequencing  lessons  to  form 
the  syllabus.  Steps  used  in  organization  and  sequencing  proceeded 
from  flight  lessons,  to  simulator  lessons,  and  finally  to  ground 
school  lessons.  The  flight  syllabus  was  formulated  first  and  pro- 
vided the  framework  around  which  the  remainder  of  the  syllabus  was 
organized  and  sequenced.  The  specification  for  the  F-4  ISO  program 
contained  guidelines  to  be  used  in  developing  the  flight  syllabus. 
These  guidelines  were  as  follows: 

1.  Determine  from  the  problem  analysis  data  the  maximum 
feasible  number  of  flights  that  can  be  contained  in  the 
syllabus. 

2.  Identify  any  content  requirements,  ordering  requirements, 

or  other  constraints  which  are  imposed  by  readiness  training 
squadrons  or  higher  authority. 

3.  Extract  from  the  objectives  hierarchy  all  objectives 
designated  during  media  selection  as  requiring  aircraft 
flight. 
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4.  Examine  the  objectives  to  identify  common  logical 
groupings. 

5.  Sort  the  pilot  and  RIO  flight  objectives  into  aircraft 
flights  within  imposed  constraints.  Further  guidelines 
for  sorting  the  objectives  are  listed  below: 

(a)  Structure  the  first  flight  so  that  it  contains 
the  absolute  minimum  number  of  objectives  required 
to  have  a flight. 

(b)  In  subsequent  flights  add  new  tasks  so  that  the 
flights  proceed  from  easier  to  more  difficult. 

(c)  For  each  flight,  group  objectives  so  that  there 
are  sufficient  objectives  to  warrant  a flight 
but  not  so  many  that  each  cannot  be  properly 
evaluated. 

(d)  Include  new  objectives  in  each  flight. 

(e)  Structure  flights  so  that  when  a new  objective 
is  introduced,  it  is  practiced  and  evaluated 
in  that  flight,  in  several  other  immediately 
following  flights,  and  then  again  at  periodic 
intervals  as  needed  to  maintain  proficiency. 

(f)  Group  objectives  into  flights  so  that  the 
prerequisite  classroom  and  trainer  periods  for 
each  flight  form  a set  of  instructional  blocks 
which  are  reasonably  unified  in  subject  matter 
and  approximately  equal  in  size. 

The  guidelines  outlined  in  1-5  above  were  closely  followed  in 
developing  the  flight  syllabus.  Based  on  problem  analysis  data, 
supplemented  by  current  figures  on  aircraft  availability,  it  was 
determined  that  the  number  of  flights  in  the  ISD  syllabus  should 
be  in  accord  with  the  number  of  flights  in  the  current  syllabus. 
Several  squadron  constraints  were  incorporated  into  the  syllabus. 
These  included  the  following: 

1.  Completion  of  the  familiarization  and  flight  support 
ground  school  and  trainer  instruction  as  specified 
in  the  Training  and  Readiness  Manual,  plus  open  and 
closed  book  exams,  prior  to  the  first  flight. 

2.  Completion  of  carrier  qualification  ground  school 
and  50  flight  hours  prior  to  carrier  qualification 
flights. 

3.  Ten  flight  hours  in  the  aircraft  prior  to  the  first 
night  flight. 
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4.  Fifteen  hours  in  the  aircraft  prior  to  firing  the 
first  air-to-air  missile. 

5.  Fifteen  hours  to  include  one  night  flight  in  the 
aircraft  prior  to  the  first  cross-country. 

Objectives  selected  for  flight  lessons  during  media  selection 
were  extracted  from  the  media  selection  report.  These  objectives 
were  examined  for  common  logical  groupings,  based  primarily  upon 
the  hierarchy  structure. 

Due  to  training  requirements  and  simulator  constraints, 
strictly  adhering  to  sorting  guidelines  (a)-(f)  above  was  found 
to  be  infeasible  in  many  cases.  For  example,  it  was  judged  to  be 
training  efficient  to  include  most  flight  objectives  for  the  air- 
to-air  phase  in  the  first  air-to-air  flight.  This  was  done  because 
the  employment  of  different  tactics  and  maneuvers  is  a function 
of  relative  aircraft  positions  and  in  practice  primarily  involves 
decision  processes  based  on  the  situation.  Subsequent  flights, 
then,  involved  the  selection  of  the  appropriate  maneuvers  and 
tactics  from  those  introduced  in  ground  school  and  in  the  f/irst 
air-to-air  flight.  Since  most  objectives  were  specified  in  the 
first  flight,  the  air-to-air  portion  of  the  syllabus  was  not 
integrated  so  that  each  flight  was  proceeded  by  ground  school  and 
simulator  lessons.  Rather  ground  school  lessons  were  grouped  at 
the  beginning  of  the  air-to-air  phase  followed  by  a mixing  of 
simulator  and  flight  lessons. 

Simulator  constraints  had  a significant  effect  on  formulating 
the  air-to-ground  portion  of  the  syllabus.  Since  the  Weapons 
System  Trainer  (WST)  has  no  visual  system,  all  air-to-ground 
practice  was  specified  for  flights.  As  with  air-to-air  training, 
air-to-ground,  ground  school  was  placed  before  the  flights.  The 
first  air-to-ground  flight  contains  most  of  the  air-to-ground 
objectives,  with  subsequent  flights  requiring  decision  processes 
based  on  the  tactical  situation.  The  grouping  of  initial  ground 
school,  followed  by  practice,  is  sound  in  terms  of  learning  theory. 
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i.e.,  provide  the  student  with  the  information  required  to  select 
among  alternatives,  then  place  him  in  problem-solving  situations 
which  require  decisions  based  upon  the  information. 

Despite  the  difficulties  encountered  in  following  the  sorting 
guidelines  for  certain  portions  of  the  syllabus,  both  the  pilot 
and  RIO  syllabi  were  generally  structured  in  accordance  with  the 
guidelines.  Starting  with  the  first  flight  which  contains  a 
minimum  number  of  basic  objectives  the  flow  proceeds  from  easier 
to  more  difficult.  Flights  contain  enough  objectives  to  warrant 
a flight,  but  not  so  many  as  to  preclude  effective  performance 
evaluation.  Practice  of  objectives  is  spaced  so  that  after  initial 
introduction  objectives  are  practiced  in  immediately  succeeding 
flights  and  periodically  thereafter.  Consistency  in  subject  matter 
content  is  maintained  from  prerequisite  ground  school,  through 
simulator  lessons,  to  flights.  This  is  the  most  fundamental 
principle  in  the  sequencing  of  instruction.  The  student  should 
first  thoroughly  know  and  understand  the  procedures  required  for 
the  performance  of  a task.  This  initial  academic  instruction 
should  be  followed  when  feasible,  by  ground-based  practice  to 
develop  task  skills.  The  task  skills  should  then  be  practiced 
and  refined  in  the  aircraft. 

Following  the  development  of  the  flight  syllabus  the 
simulator  exercises  and  ground  school  lessons  were  organized  and 
sequenced  around  the  framework  provided  by  the  flight  syllabus. 

• Guidelines  provided  in  the  specification  for  simulator  exercises 

were  as  follows: 

. 1.  Precede  each  aircraft  flight  with  a crew  coordination 

trainer  exercise  during  which  all  objectives  to  be 
practiced  during  the  flight  are  practiced  and  evaluated. 

2.  Identify  new  crew  coordination  simulator  exercises 
for  those  objectives  which  can  be  performed  in  a 
trainer  but  not  in  flight,  e.g.«  emergency  procedures. 
After  initial  practice  intersperse  these  objectives 
throughout  the  remaining  trainer  exercises. 

O 
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3.  Sequence  emergency  procedures  trainers  prior  to 
the  first  flight. 

4.  Develop  individual  position  trainer  exercises  to 
support  each  crew  simulator  exercise. 

Guidelines  for  organizing  and  sequencing  ground  school  lessons 
were  as  follows: 

1.  Use  the  objectives  hierarchies  to  identify  objectives 
that  are  prerequisite  to  the  first  trainer  exercise. 

2.  Group  objectives  into  a series  of  lessons  following 
criteria  for  good  lesson  structure,  i.e.,  subject 
matter  content,  sequencing  within  lessons,  and  length. 

3.  Use  the  objectives  hierarchies  to  identify  objectives 
which  are  prerequisites  the  other  simulator  and 
aircraft  exercises  and  group  the  objectives  into 
series  of  lessons. 

Although  the  organization  and  sequencing  of  simulator  lessons 
and  ground  school  lessons  were  specified  as  sequential  steps,  it 
was  found  that  executing  the  two  steps  in  parallel  facilitated 
overall  syllabus  development.  This  was  true  because  in  many  cases, 
for  training  efficiency,  it  was  deemed  desirable  for  simulator 
lessons  to  be  interspersed  between  ground  school  lessons  rather 
than  between  ground  school  and  flight  lessons.  In  other  cases 
ground  school  lessons  directly  preceded  flight  lessons  or  flight 
lessons  were  grouped  without  intervening  ground  school  or  simulator 
lessons.  Deviations  from  the  guidelines  for  organizing  and  sequencing 
simulator  and  ground  school  lessons  did  not  result  in  a departure 
from  the  fundamental  philosophy  of  syllabus  organization  stated 
above,  i.e.,  instruction  in  a task  should  proceed  from  academic 
instruction  to  skills  practice  in  a simulator  to  skills  practice 
in  the  aircraft. 

The  final  step  in  syllabus  development  was  to  divide  the 
sequence  of  lessons  into  units.  By  definition  a unit  which 
contains  a flight  should  end  with  that  flight,  thus  the  next 
lesson  is  the  first  lesson  of  the  next  unit.  In  the  pilot  and 
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RIO  syllabi  some  units  contain  multiple  flights.  When  no  ground 
school  or  simulator  lessons  were  scheduled  between  multiple  flights 
and  the  same  or  similar  skills  were  practiced  in  the  successive  flights, 
no  divisions  between  units  were  specified.  Unit  boundaries  should 
be  based  more  on  subject  matter  content  distinctions  rather  than 
upon  arbitrary  divisions  based  on  instructional  media. 

SME  Training.  Extensive  SME  training  was  not  required  for  syllabus 
development.  A one-day  seminar  was  conducted  to  acquaint  the  SMEs 
with  the  guidelines  to  be  used  and  to  formulate  specific  procedures 
to  be  used  in  developing  the  pilot  and  RIO  syllabi. 

Problems  and  Solutions.  Syllabus  development  was  the  most  difficult 
task  carried  out  in  this  program.  The  fundamental  problem  was  the 
lack  of  agreement  among  SMEs  on  the  optimal  organization  and 
sequencing  of  objectives.  Each  SME  had  a different  perspective  on 
how  the  pilot  and  RIO  syllabi  should  be  structured.  This  led  to 
lengthy  discussions  which  produced  little  progress.  In  order  t 6 
facilitate  the  development  process,  the  recently  revised  VMFAT-lOl 
Training  and  Readiness  (T  and  R)  Manual  was  used  to  provide 
guidance.  The  syllabus  outlined  in  the  T and  R manual  basically 
follows  the  rules  of  good  syllabus  structure.  By  usir.g  it  as  a 
guide,  consensus  among  the  SMEs  was  achieved,  and  syllabus  develop- 
ment proceeded.  The  SBOs  from  hierarchy  and  SBO  development  were 
organized  and  sequenced  using  the  T and  R syllabus  as  a general 
guide. 

An  anticipated  problem  of  any  ISD  program  is  FRS  acceptance 
of  the  finished  product.  By  staying  within  the  general  framework 
of  the  T and  R syllabus,  the  probability  of  FRS  rejection  and/or 
requirements  for  major  modifications,  at  least  at  the  syllabus 
level  of  training  course  development,  was  reduced.  This  was  a 
desirable  by-product  of  the  approach  adopted  rather  than  a goal. 
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The  hierarchies  and  SBOs  were  mission  oriented  rather  than 
training  oriented.  As  a consequence  many  objectives  which  the 
SMEs  considered  essential  for  training  were  not  included  in  the 
hierarchies.  The  most  significant  subject  matter  area  for  which 
SBOs  were  not  contained  in  the  hierarchies  was  systems  information. 
Although  training  courses  developed  using  ISO  techniques  do  not 
generally  include  detailed  systems  information,  it  is  still  essential 
that  certain  background  systems  information,  in  addition  to  that 
which  can  be  gleaned  from  the  mission-oriented  SBOs,  be  taught. 

Other  objectives  which  were  not  included  in  the  hierarchies,  but 
Which  are  an  important  part  of  training,  were  aircraft  maneuvers, 

Such  as  touch  and  go  landings  and  timed  turns.  Objectives  to  cover 
systems  information  and  training  maneuvers  were  included  in  lessons 
as  the  syllabus  was  developed. 

Throughout  syllabus  development,  objectives  not  contained  in 
the  hierarchies  were  included  in  lessons.  This  is  to  be  expected 
since  ISO  is  a highly  iterative  process,  i.e.,  as  the  process 
proceeds  from  task  to  task  new  insights  are  developed  which  improve 
the  end  products  of  previous  tasks.  As  the  SBOs  were  structured 
into  lessons  and  the  lessons  were  reviewed  for  completeness, 
objectives  were  added  to  provide  complete  instruction  in  certain 
topic  areas.  The  addition  of  these  supplemental  objectives,  plus 
the  objectives  for  systems  and  training  maneuvers,  will  enhance 
the  quality  of  the  pilot  and  RIO  training  courses. 

Since  media  selection  preceded  syllabus  development,  in  the 
ISD  specification,  it  was  implicit,  although  not  directly  stated, 
that  media  decisions  should  affect  the  grouping  of  objectives 
into  lessons.  This  was  true  for  the  flight  and  simulator  portions 
of  the  syllabus.  The  start  point  for  these  portions  was  a listing 
of  all  objectives  which  required  simulators  and/or  flights.  The 
organization  of  ground  school  lessons,  however,  was  done  on  the 
basis  of  subject  matter  content  and  hierarchy  structures  without 
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regard  for  selected  media,  i.e.,  the  overriding  criterion  for 
good  lesson  structure  is  appropriateness  and  completeness  of 
content. 

Most  of  the  ground  school  lessons  contained  objectives  for 
which  the  same  medium  had  been  selected.  Other  lessons,  however, 
contained  more  than  one  medium  across  the  objectives.  When  this 
happened,  each  lesson  was  examined  to  determine  which  medium  was 
the  most  frequently  selected  for  the  objectives  in  that  lesson. 

Media  selected  for  the  remaining  objectives  in  the  lesson  were 
then  examined  to  determine  if  one  of  the  alternate  media  from 
the  media  selection  model  was  the  same  as  the  predominant  medium 
in  the  lesson.  In  most  cases,  matches  were  found  and  the  pre- 
dominant medium  was  selected  for  all  objectives  in  the  lesson. 

In  those  instances  in  which  no  alternate  medium  matched,  the 
content  of  the  objective  was  examined  to  determine  whether  or 
not  the  objective  should  be  repcsitioned  in  the  syllabus.  No 
objectives  were  repositioned.  For  each  case  the  content  and 
context  requirements  of  the  objective  were  judged  to  be  more 
important  than  media  considerations. 

Contract  time  allotted  for  syllabus  development  was  only 
four  weeks.  This  was  an  unrealistically  short  period  for  the 
amount  of  work  required  to  structure  pilot  and  RIO  syllabi  for 
both  the  Navy  and  Marines  and  to  identify  differences  between  the 
J and  N aircraft  models.  In  addition,  SME  support  was  less  than 
that  needed  for  the  task.  As  a result,  14  weeks  were  required 
to  complete  syllabus  development. 

Although  validation  of  the  syllabus  by  the  three  FRSs  at 
which  the  training  courses  will  be  implemented  was  not  contractually 
required,  it  was  anticipated  that  this  could  be  accomplished  with- 
in the  scope  and  time  constraints  of  the  contract.  Due  to  the 
delay  in  completion,  however,  it  was  not  feasible  to  perform  a 
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complete  validation.  Since  lesson  specification  development 
started  immediately  after  syllabus  completion  under  another 
contract,  it  was  decided  that  validation  could  be  carried  out 
concurrently  with  the  early  parts  of  lesson  specification  develop- 
ment. 

TRAINING  SUPPORT  REQUIREMENTS  ANALYSIS. 

Overview.  The  final  task  was  to  perform  a Training  Support  Re- 
quirements Analysis  (TSRA).  The  purpose  of  the  TSRA  was  to 
estimate  the  costs  of  personnel,  equipment,  materials,  and  services 
which  will  be  required  to  further  design,  develop,  implement,  and 
maintain  the  pilot  and  RIO  training  courses.  This  information  will 
be  used  for  future  planning  and  budgeting.  The  detailed  analysis 
is  contained  in  the  Training  Support  Requirements  Report. 

During  the  design  phase  lesson  specifications  will  be  produced. 
Since  this  phase  had  been  contracted  prior  to  the  development  of 
the  TSRA,  no  cost  estimates  were  made.  Estimates  for  all  other 
future  phases  of  the  F-4  ISD  program  were  provided  using  the  pro- 
cedures outlined  in  the  following  section. 

Procedures . The  development  phase,  which  consists  of  the  authoring 
and  production  of  course  materials,  is  the  single  most  expensive 
phase  of  an  ISO  program.  Two  basic  sources  of  information  were 
used  in  the  calculation  of  development  cost  estimates:  (1)  syllabus 
parameters  on  the  number  of  lessons  in  each  instructional  medium 
and  (2)  personnel,  materials,  and  service  costs  required  to  produce 
lessons  in  each  medium.  By  merging  these  two  sources  of  information, 
total  estimated  costs  for  production  were  determined. 

Syllabus  parameters  were  determined  by  totaling  the  number  of 
lessons  in  each  medium.  This  data  is  shown  in  Table  5.  Personnel, 
material,  and  service  costs  for  the  production  of  one  lesson  in 
each  medium  were  assimilated  using  the  format  shown  in  Figure  10. 
Appendix  D contains  development  costs  for  each  of  the  media  selected 
for  use  in  the  F-4  training  course. 
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INSTRUCTIONAL 

MEDIUM 


Ground  School 
Slide  Tape 
Random  Access  Slide 
Workbook 
Videotape 

Mediated  Interactive  Lecture 
Simulator 
Aircraft 


NO.  OF 
LESSONS 


77 

12 

2 

1 

3 

89 

293 


Figure  5.  Number  of  Lessons  in  Each 
Instructional  Medium 
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Summarizing  across  categories  within  media  provided  total  costs 
per  medium.  The  total  costs  per  medium  were  then  summed  to  give 
a cost  estimate  for  the  production  of  one  complete  set  of  course 
materials. 

Since  it  was  understood  that  all  reproduction  of  materials 
would  be  done  by  the  government,  no  reproduction  costs  were 
estimated.  Estimates  of  the  number  of  sets  of  consummable  and 
non-consummable  materials  were  provided  to  assist  the  government 
in  estimating  reproduction  costs. 

In  addition  to  production  costs,  facilities  requirements  for 
the  production  phase  were  estimated.  Assuming  production  on-site 
at  MCAS,  Yuma,  it  was  determined  that  the  facilities  available  in 
the  MCCTRG-10  ISD  Department  are  sufficient  to  support  the  pro- 
duction effort.  Since  it  was  assumed  that  any  company  contracted 
for  production  would  furnish  its  own  equipment,  equipment  costs 
for  production  were  not  estimated. 

The  primary  cost  of  implementation  will  be  the  acquisition  of 
carrels  and  components  required  for  the  presentation  of  slide-tape 
lessons.  Determination  of  the  number  of  carrels  required  at  each 
Fleet  Readiness  Squadron  (FRS)  and  each  operational  and  reserve 
squadron  that  will  use  the  training  materials  was  based  on  the 
number  of  lessons  that  will  be  presented  via  slide-tapes,  the 
length  of  the  training  course,  and  the  projected  student  flow 
rate.  The  costs  of  fully  equipped  carrels  were  based  upon  current 
GSA  prices  for  standard  equipment  purchased  by  the  Navy  and  in- 
cluded a factor  to  allow  for  inflation  during  the  interval  before 
the  equipment  is  purchased. 

Facilities  required  to  house  the  learning  centers  and  to 
provide  space  for  materials  storage,  lectures,  and  briefings  were 
estimated.  The  results  of  a preliminary  survey  of  the  three  FRSs 
indicated  that  spaces  are  available  to  satisfy  the  requirements  of 
the  new  training  courses;  however,  some  modifications  will  be 
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required  in  setting  up  the  learning  centers.  These  include  in- 
stallation of  carpeting,  supplemental  electrical  power,  and  in- 
stallation of  shelves  for  materials'  storage.  Due  to  the  large 
number  of  potential  users  of  the  training  materials  in  the  active 
and  reserve  communities,  it  was  not  feasible  to  survey  available 
facilities  at  the  numerous  locations.  Since  space  requirements 
will  be  minimal,  it  is  conjectured  that  adequate  facilities  are 
available  at  most  locations. 

FRS  instructor  requirements  for  implementation  were  estimated. 
This  estimate  was  based  on  the  projected  student  flow  rate  through 
the  training  courses,  the  number  of  students  that  will  be  in 
training  at  any  one  time  as  derived  from  a schedule  of  overlapping 
classes,  and  the  number  and  projected  utilization  of  media  that 
require  instructors.  Included  in  the  estimate  was  a factor  to 
account  for  leave,  TAD,  etc. 

The  final  step  in  the  TSRA  was  to  estimate  resource  require- 
ments for  five-year  maintenance  and  evaluation  of  the  training 
courses.  Revision,  maintenance,  and  quality  control  activities 
will  be  most  extensive  during  the  first  year  of  the  five-year  period 
Total  requirements  for  the  first  year  were  determined  by  multiplying 
the  personnel,  materials,  and  services  directly  tied  to  initial 
courseware  development  and  production  by  .35.  Annual  requirements 
for  the  next  four  years  were  determined  by  applying  a factor  of 
.20. 

The  TSRA  results  indicated  a costly  development,  implementation 
and  maintenance  program  for  the  F-4  pilot  and  RIO  courses.  Total 
estimated  costs  for  each  phase  were  as  follows: 


Development 

Implementation 

Five-year  evaluation 
and  maintenance 

Year  1 

Years  2-5 

Grand  Total 


$ 603,047 
118,266 


211,066 

482,436 

$1,414,815 


NAVTRAEQU I PCEN  77-C-0081-1 


The  Training  Support  Requirements  Report  contained  infor- 
mation sufficient  to  make  cost  trade-offs  among  the  ground  school 
media.  Costs  for  each  medium  could  be  compared  to  determine 
relative  costs.  When  used  in  conjunction  with  a graphical  flow 
chart  of  the  syllabus,  which  was  part  of  the  TSRA,  the  costs  could 
be  related  to  specific  lessons,  and  potential  sources  of  savings 
could  be  identified. 

SHE  Training.  Since  the  TSRA  was  a contractor  task,  no  formal 
training  was  conducted.  A two-hour  seminar  was  held  to  acquaint 
the  SMEs  with  the  contents,  purposes,  and  applications  of  the  TSRA. 

Problems  and  Solutions.  Formulation  of  the  TSRA  went  smoothly  with 
no  major  problems.  Several  small  problems,  however,  developed 
during  the  course  of  the  analysis.  These  will  be  discussed  below. 

Three  production  companies  were  surveyed  to  collect  the  data 
required  to  complete  Figure  10.  Among  the  companies  there  was  a 
lack  of  consistency  in  interpretation  of  the  roles  and  relative 
efforts  of  the  different  labor  categories.  This  was  particularly 
true  for  graphics  specialist  and  paste-up  artist.  One  company 
defined  a paste-up  artist  as  one  who  could  also  do  simple  illus- 
trations, with  the  graphics  specialist  required  for  complex  illus- 
trations. Another  company,  however,  assigned  all  illustrations  to 
the  graphics  specialist,  with  the  paste-up  artist  serving  a 
supporting  role  for  paste-up  work  only.  In  addition  to  discrepancies 
in  personnel  roles  and  responsibilities,  companies  differed  in  hourly 
wage  rates.  This  is  due  to  different  base  rates  across  companies, 
as  well  as,  different  overhead  rates. 

The  values  which  were  entered  in  the  cells  in  Figure  10  and 
the  wage  rates  used  as  multipliers  for  the  personnel  hours  are 
averages  across  the  three  production  companies.  The  estimates  of 
materials  development  and  revision  costs  are,  therefore,  considered 
to  be  representative  of  costs  across  the  industry.  Estimates  from 
any  given  company  would  be  expected  to  differ  somewhat  from  those 
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programs.  The  selection  of  lessons  to  be  mediated  should  be 
based  on  the  general  characteristics  of  lesson  content,  such  as 
update  requirements,  need  for  visuals,  etc.  The  content  of  all 
lessons,  however,  should  be  as  specified  during  the  ISD  program. 

TRAINING  PROGRAM  SUMMARY. 

The  structure  of  the  proposed  training  courses  is  embodied 
in  the  pilot  and  RIO  syllabi.  The  syllabi,  rather  than  being 
"concrete"  products,  constitute  a working  document  which  will  be 
revised  during  subsequent  phases  of  the  ISO  program.  Due  to  the 
overlap  of  this  contract  with  a subsequent  contract  for  the 
development  of  lesson  specifications,  syllabus  revisions  are 
occurring  during  the  performance  period  for  this  contract.  Based 
on  a preliminary  evaluation  of  the  revisions  being  incorporated 
by  the  SMEs,  it  is  not  feasible  to  discuss  in  detail  syllabus 
parameters  such  as  length,  number  of  lessons  per  medium,  etc. 
Rather,  an  overview  in  terms  of  media  types,  instructional 
strategies,  and  utilization  of  trainers  and  the  aircraft  will 
be  provided. 

The  candidate  media  were  specified  in  the  media  selection 
model;  thus  the  syllabus  contains  only  those  media.  They  are 
mediated  interactive  lecture,  workbook,  slide-tape,  videotape, 
simulator,  and  aircraft. 

The  guidelines  for  syllabus  development  indicated  that  the 
number  of  flights  should  be  maximized  consistent  with  aircraft 
availability  and  projected  student  flow  rate.  The  ISD  syllabus 
contains  essentially  the  same  number  of  flights  as  the  current 
syllabus.  It  is  anticipated  that  this  will  not  change  during 
revisions  in  subsequent  phases  of  the  program. 

The  primary  innovation  in  the  proposed  syllabus  is  the  in- 
dividualization of  instruction  as  opposed  to  the  current 
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“lock  step",  classroom  orientation.  The  system  is  individualized 
in  the  sense  that  most  ground  school  lessons  are  presented  in 
either  a slide-tape  carrel  or  in  a workbook  with  a test  at  the 
end  of  each  lesson.  Students  can  proceed  at  their  own  pace 
within  lessons.  Across  lessons  it  is  anticipated  that  self-pacing 
will  be  feasible  within  the  constraints  of  equipment  availability 
and  prerequisite  requirements. 

Self-pacing  combined  with  the  more  efficient  presentation  of 
information  in  the  individualized  media  should  in  theory  yield  a 
training  program  whose  duration  is  less  than  that  of  non-mediated, 
paced  programs.  Due  to  external  constraints  such  as  aircraft  and 
trainer  availability  and  additional  duties  required  of  students, 
current  projections  are  that  this  will  not  be  the  case  for  the 
new  F-4  pilot  and  RIO  courses.  Typical  completion  time  for  the 
Marine  60  percent  portion  and  the  Navy  80  percent  portion  should 
be  about  six  months.  Since  the  remaining  40  percent  and  20  percent 
respectively  are  taught  at  operational  squadrons  where  training  is 
interspersed  with  normal  operations,  no  estimate  of  duration  for 
these  portions  can  be  made. 

The  training  courses  outlined  in  the  pilot  and  RIO  syllabi 
adhere  to  the  principles  of  good  syllabus  structure.  That  is, 
lessons  are  sequenced  so  as  to  optimize  student  learning  and 
retention  by  integrating  ground  school  instruction  with  "hands-on" 
practice  in  the  trainers  and  the  aircraft.  Within  lessons  segments 
are  organized  to  enhance  instructional  flow  within  content  areas. 

It  is  anticipated  that,  when  implemented,  the  pilot  and  RIO  courses 
will  produce  better  qualified,  more-standardized  F-4  aircrewmen. 
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Results  in  terms  of  the  structure  of  the  pilot  and  RIO 
syllabi  were  presented  in  the  previous  section.  In  this  section, 
results  and  conclusions  relevant  to  the  ISD  model  used  in  the 
program  and  to  ISD  in  general  will  be  presented  under  five  different 
headings: 

0 ISD  Methods 
0 ISD  Implementation 
0 ISD  Resources 
0 Documentation  Adjustments 
0 Speculative  Observations 

ISD  METHODS 

The  ISD  model  used  in  this  program  is  the  most  prescriptive 
model  available  in  terms  of  the  specification  of  discrete  tasks 
required  to  produce  an  ISD-based  training  program.  Within  the 
tasks  many  of  the  procedures  contain  precisely  delineated  steps. 

The  goal  of  "mechanizing"  ISD  into  a straight-forward  set  of  pro- 
cedures which  can  be  executed  by  "turning  the  crank",  has  been 
pursued  by  training  technologists  for  many  years.  In  the  6.2 
model  this  goal  has  been,  at  least,  partially  achieved. 

Since  the  procedures  detailed  in  the  model  are  highly  structured 
relative  to  other  approaches  in  which  broad  guidelines  rather  than 
detailed  procedures  are  provided,  it  is  easy  for  a training  tech- 
nologist first  applying  the  model  to  implement  the  procedures 
without  considering  the  intangible,  judgment  factors  which  are 
integral  components  of  ISD.  This  is  not  a criticism  of  the  goal 
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of  procedural i zing  ISD.  Rather,  it  is  a caution  against  taking 
the  procedures  at  face  value  without  considering  the  rationale 
behind  the  steps.  In  studying  the  procedures  in  light  of  general 
principles  of  learning  and  real  world  constraints,  insights  will 
be  gained  which  assist  in  integrating  the  intangible  factors  into 
training  program  development.  Also,  additional  general  procedures 
and  others  which  are  particularly  relevant  to  the  system  under 
study  will  be  derived. 

In  the  following  paragraphs,  conclusions  and  recommendations 
pertaining  to  each  major  task  carried  out  in  the  F-4  ISD  program 
will  be  discussed.  Some  of  the  points  were  considered  in  Section 
II,  Implementation,  while  others  are  presented  for  the  first  time 
in  this  section. 

OBJECTIVES  HIERARCHIES  AND  SBOs.  The  task  of  developing  objectives 
hierarchies  and  SBOs  went  smoothly.  The  SMEs  understood  the  tech- 
niques and  implemented  them  well.  One  deficiency  in  the  procedures 
outlined  in  the  model  became  evident  during  the  development  process. 
This  was  the  lack  of  clearly  defined  guidelines  for  determining 
the  level  in  the  hierarchies  above  which  detail  was  not  sufficient 
and  below  which  additional  detail  was  trivial.  The  source  of  the 
deficiency  in  the  specification  was  the  failure  to  relate  objectives 
to  the  way  in  which  they  will  be  used  during  subsequent  ISD  tasks, 
i.e.,  as  segments  of  instruction.  In  order  to  remedy  the  deficiency 
it  is  recommended  that  a description  of  how  objectives  will  be 
converted  into  segments  of  instruction  be  included  in  the  instructions 
for  hierarchy  and  SBO  development.  As  part  of  this  description, 
guidelines  should  be  provided  which  define  the  scope  of  a segment 
of  instruction,  e.g.,  two  to  five  minutes  of  ground  school  in- 
struction for  which  four  to  eight  supporting  visuals  will  be 
required;  for  manipulative  tasks,  an  action  which  can  be  evaluated 
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and  would  be  expected  to  be  practiced  from  two  to  five  minutes 
during  a simulator  or  flight  lesson.  By  providing  such  guidelines 
the  level  of  detail  in  the  hierarchies  will  be  more  uniform. 

More  importantly,  however,  this  consistency  will  contribute  to 
more  efficient  syllabus  organization  and  to  less  revision  during 
the  development  of  lesson  specifications. 

The  examples  presented  above  are  not  intended  to  be  specific 
guidelines.  They  are  based  on  limited  experience  gained  during 
the  early  stages  of  lesson  specification  development.  Datd  from 
this  and  other  programs  should  be  gathered  and  analyzed  to  deter- 
mine appropriate  parameters  which  define  a typical  segment  of 
instruction. 

The  conditions  and  standards  portions  of  each  objective 
serve  no  function  during  hierarchy  development.  They  may  serve 
some  function  during  media  selection,  given  a selection  model 
into  which  they  are  incorporated.  At  this  point  in  ISO  model 
formulation,  it  is  not  clear  how  they  would  be  used  for  media 
selection.  Certainly  the  model  specified  for  the  F-4  ISO  program 
did  not  utilize  conditions  and  standards. 

To  formulate  conditions  and  standards  well  is  time  consuming 
for  the  SMEs  and  detracts  from  the  primary  task  of  identifying 
and  structuring  training  objectives.  Even  well-thought  out  con- 
ditions and  standards  are  not  optimal  at  the  hierarchy  stage  of 
training  system  development.  Such  conditions  and  standards  can 
be  specified  only  after  media  have  been  selected  and  testing 
philosophies  have  been  established.  In  the  present  ISD  model 
media  selection  follows  hierarchy  development,  with  the  formulation 
of  testing  philosophies  taking  place  during  lesson  specification 
development.  It  is  recommended  that  the  requirements  for 
conditions  and  standards  during  SBO  development  be  eliminated. 

This  will  allow  hierarchy  development  to  proceed  more  quickly  and 
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will  allow  the  SMEs  to  concentrate  on  their  expertise,  knowledge 
of  the  aircraft.  Detailed  conditions  and  standards  would  continue 
to  be  formulated  as  part  of  lesson  specifications  as  outlined  in 
the  ISD  model . 

A primary  deficiency  in  the  F-4  hierarchies  is  the  lack  of 
objectives  which  cover  systems  information  and  training-only  objectives. 
This  resulted  from  the  mission  orientation  of  the  task  listing. 

When  the  task  listing  was  expanded  into  hierarchies  and  SBOs,  the 
mission  orientation  was  maintained.  Some  systems  information, 
parti cul ary  emergencies,  is  contained  in  the  hierarchies;  however, 
the  amount  and  detail  of  the  information  is  significantly  less 
than  that  which  is  acceptable  to  the  F-4  training  community.  To 
remedy  this  situation  systems  objectives  were  added  during  syllabus 
development.  Efforts  were  made  to  insure  that  the  supplemental 
objectives  were  relevant  to  mission  performance  and  were  not 
tangential  from  an  ISD  point  of  view,  but  were  sufficiently  de- 
tailed to  satisfy  the  training  community. 

Training  objectives  were  also  added  during  syllabus  development. 
These  were  primarily  performance  objectives  which  are  required  in 
the  training  environment  but  which  would  not  be  reflected  in 
mission-oriented  SBOs.  Examples  of  these  objectives  are  touch 
and  go  landings  and  timed  turns. 

It  is  recommended  that  systems  and  training-only  objectives  be 
identified  during  the  development  of  hierarchies  and  SBOs.  Most 
of  these  objectives  can  be  incorporated  into  the  hierarchies  as 
prerequisites  to  objectives  identified  as  mission  objectives. 

Others  may  not  clearly  relate  to  the  mission  objectives  and  will, 
therefore,  constitute  a second  category  of  objectives.  The 
fundamental  concern  is  that  the  identification  and  structuring  of 
objectives  be  as  complete  and  accurate  as  possible,  at  as  early 
a stage  in  the  ISD  process  as  possible. 
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MEDIA  SELECTION.  The  media  selection  model  specified  for  use  in 
the  F-4  ISD  program  is  not  required  in  the  most  recent  specifi- 
cation of  training  requirements  for  aviation  weapons  systemfc, 

MIL-T-29053.  Rather,  the  contractor  is  given  the  latitude  to 
select  a model  within  certain  constraints,  subject  to  contract 
technical  monitor  approval.  The  current  model  is  presented  as 
one  alternative  which  satisfies  the  specified  constraints.  This 
latitude  afforded  future  ISD  contractors  provides  an  indication 
that  NAVTRAEQUIPCEN  ISD  personnel  view  media  selection  as  a part 
of  the  ISD  model  which  needs  improvement. 

This  contractor  has  not  formulated  a media  selection  mddel 
which  is  any  more  prescriptive  than  the  F-4  model  or  the  numerous 
other  models  described  in  the  literature.  There  are,  however, 
several  recommendations  which  will  improve  the  model  used  in  the 
F-4  program.  More  precise  definitions  of  the  questions  and 
responses  need  to  be  included  in  the  description  of  how  to  use  | 

the  model.  These  definitions  should  be  accompanied  by  a number 
of  worked  examples  which  span  the  range  of  types  of  objectives 
which  will  be  traced  through  the  media  map. 

The  number  of  candidate  media  identified  in  the  model  should 
be  broadened  to  reflect  other  feasible  devices  which  are  available. 

Since  some  media  are  similar  in  terms  of  instructional  capabilities, 
these  can  be  grouped  into  media  classes  with  further  differentiation 
based  on  cost  and  dependability. 

If  the  F-4  media  model  is  used  in  future  programs,  it  is 
recommended  that  delivery  of  multiple  copies  of  a completed  media 
map  for  each  objective  not  be  required.  Since  media  selection  is 
altered  somewhat  during  syllabus  development  and  since  the  size 
of  deliverable  volumes  is  prohibitive,  media  selection  should  be 
considered  an  interim  step  not  requiring  a formal  deliverable 
end-product.  Media  decisions  will  be  summarized  after  the  syllabus 
has  been  developed. 
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Finally,  media  selection  should  be  a part  of  syllabus 
development  rather  than  a preceding  step.  Performance  objectives 
need  to  be  identified  so  that  simulator  and  flight  lessons  can 
be  organized.  Ground  school  media,  however,  need  not  be  selected 
until  after  the  ground  school  lessons  have  been  organized.  These 
lessons  should  be  structured  on  the  basis  of  subject  matter  content, 
not  on  the  basis  of  media  selected  for  each  objective.  This  pro- 
cedure will  eliminate  the  duplication  of  effort  in  which  media  are 
selected  for  each  objective  but  some  of  which  are  changed  as  a 
result  of  syllabus  organization. 

SYLLABUS.  The  guidelines  for  syllabus  development  provided  in  the 
specification  are  good.  They  are  idealized  and  cannot  always  be 
strictly  followed  due  to  external  constraints,  such  as  training 
policies  and  simulator  capabilities.  They  are,  however,  quite 
workable  within  the  unique  constraints  of  different  training  situations. 

Several  recommendations  have  resulted  from  use  of  the  syllabus 
development  procedures  in  the  F-4  ISD  program.  The  specification 
that  $he  syllabus  should  contain  the  maximum  feasible  number  of 
flights  is  not  in  accord  with  traditional  ISD  philosophy.  In 
most  cases  the  maximum  feasible  number  of  flights  will  be  the 
number  in  the  existing  syllabus.  One  goal  of  ISD  is  to  decrease 
flight  time  while  increasing  the  quality  of  training  provided. 

The  nqmber  of  flights  should  be  based  upon  the  number  of  objectives 
which  require  flight  instruction  and  practice,  and  the  amount  of 
practice  required  to  master  the  objectives.  Admittedly,  this 
guideline  is  not  stated  in  operational  terms  such  as  the  existing 
guidelines.  It  is,  however,  more  realistic  and  consistent  with 
the  goals  of  ISD. 

The  specification  states  that  flights  should  proceed  from 
"easier  to  more  difficult"  by  adding  new  objectives  to  each  flight. 
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Taken  literally  this  statement  can  be  misleading.  As  new  objectives 
are  added,  flights  become  more  complex,  but  not  necessarily  more 
difficult.  Students  may  have  mastered  earlier  objectives  so  that 
the  addition  of  new  objectives  does  not  increase  difficulty.  It 
is  recommended  that  "easier  to  more  difficult"  be  changed  to 
"simpler  to  more  complex."  In  addition,  it  should  be  recbgnized 
that  within  the  total  syllabus,  a flight  to  flight  increase  in 
complexity  may  not  occur,  even  though  the  general  flow  is  from 
simpler  to  more  complex.  For  example,  in  the  F-4  pilot  s>llabus, 
air-to-air  training  precedes  air-to-ground  training.  The  final 
flight  in  the  air-to-air  portion  of  the  syllabus  is  "more  complex" 
than  the  first  flight  in  the  air-to-ground  portion. 

One  of  the  syllabus  guidelines  states  that  a flight  lesson 
shall  be  the  last  lesson  in  a unit  which  contains  a flight  and 
that  the  next  lesson  is  by  definition  the  first  lesson  in  the 
next  unit.  This  is  often  the  case.  In  other  instances,  however, 
it  is  more  logical  for  units  to  contain  multiple  flights  If  the 
objectives  to  be  practiced  in  the  flights  are  essentially  the  same 
and  if  there  is  no  ground  school  or  simulator  instruction  separating 
the  flights.  If  subject  matter  content  is  uniform,  including 
multiple  flights  in  the  same  unit  when  there  is  intervening  ground- 
based  instruction  should  also  be  allowed.  This  will  enhance 
continuity  within  units  and  provide  more  flexibility  to  structure 
units  of  approximately  equal  size. 

The  problem  in  syllabus  development  encountered  in  this  pro- 
gram (See  Section  II,  Implementation)  did  not  result  from  a basic 
deficiency  in  the  specification  being  followed.  The  process  of 
developing  a syllabus  requires  a large  judgment  component  within 
the  specified  development  guidelines.  Judgment  varies  across 
individuals,  thus,  the  SMEs  had  difficulty  agreeing  on  the  best 
way  to  structure  the  syllabus.  This  is  to  be  expected  in  any 
ISD  program,  particularly  in  one  for  a system  which  has  been  in  the 
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inventory  for  a number  of  years. 

TRAINING  SUPPORT  REQUIREMENTS  ANALYSIS.  The  format  provided  in  the 
specification  for  collecting  production  personnel  and  cost  data 
was  very  useful.  The  personnel,  materials,  and  services  categories 
were  Inclusive  for  the  media  specified  in  the  media  selection  model. 
Other  costs,  however,  were  not  specified  in  sufficient  detail  or 
were  not  mentioned  at  all.  Also,  the  funding  sources  for  the 
different  costs  need  to  be  identified.  This  enables  contracting 
personnel  to  differentiate  between  costs  which  will  come  from  ISD 
resources  and  those  which  will  come  from  other  resources. 

Equipment  categories  should  be  more  precisely  identified. 

This  will  assist  in  more  accurate  and  complete  costing  and  will 
allow  for  distinctions  between  government-furnished  and  contractor- 
furnished  equipment.  For  example,  categories  of  equipment  required 
during  production  include  materials  production  gear,  reproduction, 
and  typewriters  and  composers.  Similarly,  a breakdown  of 
facilities  in  terms  of  intended  use  is  desirable. 

In  the  specification,  no  cost  category  is  included  for  the 
formulation  and  production  of  implementation  and  quality  control 
plans  and  an  instructor  training  course.  These  are  important 
parts  of  the  production  phase  and  should  be  included  in  the  pro- 
duction cost  estimates.  Also,  costs  incurred  by  the  contracting 
agency  should  be  included. 

Revision  estimates  for  the  first  year  after  implementation  are 
determined  by  applying  a factor  of  .35  to  materials  production  costs. 
A factor  of  .20  is  used  for  each  of  the  other  out  years.  The 
limited  data  from  other  ISD  programs  suggests  that  a more  accurate 
factor  may  be  .20  for  the  first  year  and  some  lesser  factor  for 
subsequent  years.  More  data  needs  to  be  collected  to  determine 
the  appropriate  factors  to  be  used  in  revision  cost  estimates. 

so 
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ISD  IMPLEMENTATION 

Implementation  of  the  ISD  model  was  successful  and  occurred 
as  planned  with  only  minor  deviations  as  discussed  in  other  sections 
of  this  report.  There  are,  however,  three  general  issues  cdncerning 
the  manner  in  which  ISD  is  implemented.  The  first  issue,  contractor 
versus  military  SMEs,  did  not  arise  during  this  program.  The  last 
two  issues,  the  quantity  and  quality  of  military  SME  support  and 
end  product  acceptance  by  the  training  community  were  issues  raised 
in  this  program.  All  three  issues  will  be  discussed  below. 

SOURCE  OF  SMEs.  Other  ISD  programs  have  used  contractor  SMEs 
exclusively,  military  SMEs.  exclusively,  or  combinations  of  both. 
Arguments  for  the  two  extremes  essentially  boil  down  to  quarltity 
and  quality  of  work  produced  versus  latest  system  knowledge  and 
program  acceptance.  The  limited  data  available  indicate  that 
contractor  SMEs  can  be  expected  to  produce  more  and  better  dnd 
products  than  military  SMEs  in  a given  time  period.  On  the  Other 
hand,  military  SMEs  are  more  likely  to  be  versed  in  current  System 
operating  procedures  and  tactics.  In  addition,  training  programs 
produced  by  military  SMEs  are  more  likely  to  be  accepted  thah 
those  produced  by  contractor  SMEs,  i.e.»  if  the  producer  and  user 
are  one  in  the  same,  there  is  a higher  probability  of  acceptance. 

The  selection  of  an  option  cannot  be  based  solely  on  a trade- 
off of  the  above  factors.  Also  to  be  considered  are  qualifications 
of  military  SMEs  who  will  be  assigned,  the  level  of  commitment 
once  assigned,  and  the  ISD  phase  to  which  they  will  be  assigned. 

The  former  two  considerations  will  be  discussed  in  the  following 
section  on  the  structuring  of  a team  of  military  SMEs. 

The  effect  of  ISD  phases  is  a function  of  the  different 
types  of  input  required  during  the  different  phases.  The  basic 
structure  and  subject  matter  of  the  training  program  is  determined 
during  the  analysis  and  design  phases.  In  order  to  insure  that 
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the  outputs  of  these  phases  are  technically  correct  and  are 
acceptable  to  the  training  community,  it  is  important  that 
military  SMEs  be  assigned.  They  may  be  supplemented  by  con- 
tractor SMEs  to  improve  productivity,  but  must  be  heavily  in- 
volved. 

During  development  of  training  materials  a heavier  emphasis 
may  be  placed  on  the  use  of  contractor  SMEs/authors , who  are 
well -schooled  in  production  procedures.  The  contractor  SMEs 
will  refine  the  products  of  the  earlier  phases  but  will  make 
no  substantive  changes  without  input  from  military  SMEs.  By 
recognizing  this  division  the  capabilities  of  the  different 
personnel  will  be  more  effectively  utilized. 

ASSIGNMENT  OF  SMEs.  It  is  desirable  for  SMEs  assigned  to  ISD 
programs  to  be  instructors  in  a training  squadron  or  to  have  had 
recent  instructor  experience.  These  personnel  can  provide 
greater  knowledge  of  squadron  training  policies  and  general  training 
techniques  than  can  those  without  instructor  experience.  In 
addition,  their  close  association  with  the  training  community 
enhances  the  probability  of  squadron  acceptance. 

A recurring  problem  in  ISD  programs  is  the  failure  to  assign 
enough  SMEs  or  to  assign  SMEs  whose  other  duties  significantly 
detract  from  their  contribution  to  the  ISD  program.  Ideally,  an 
ISD  department  should  be  formed  within  the  training  squadron. 

SMEs  should  be  assigned  full  time  to  the  program  with  other  duties 
reduced  to  an  absolute  minimum.  Assignment  to  ISD  should  be  on 
a long  term  basis  to  maintain  stability  and  continuity  within  the 
ISD  team.  This  would  reduce  turnover  and  the  resultant  in- 
efficiencies which  have  caused  problems  in  previous  ISD  programs. 

The  ISO  department  must  have  a strong  leader  who  is 
- . ' we  tc  the  ISD  requirements  and  who  is  willing  to  work 

•’th  the  contractor.  Since  ISD  is  relatively  new  and  is 
.mi*  , . n the  mainstream  of  training  activity,  there  is  a 


I V 


Hm 


NAVTRAEQUIPCEN  77-C-0081-1 


tendency  for  looseness  in  the  ISD  team.  This  leads  to  wdrk  in- 
efficiencies, lack  of  interest,  and  waste  of  time. 

A related  problem  is  the  status  accorded  ISD  and  the  SMEs 
involved  in  an  ISD  program.  In  general,  ISD  assignments  are  not 
considered  career-enhancing  and  are  not  given  as  rewards  for  a 
good  record  of  military  performance  or  good  performance  ih  the 
weapons  sytem.  Such  a situation  can  lead  to  morale  problems  and 
low  productivity. 

A summary  of  considerations  to  be  implemented  in  the  formation 
of  military  SMEs  into  an  ISD  team,  the  structuring  of  the  team,  and 
the  status  of  the  team  is  as  follows: 

0 Make  the  ISD  team  an  integral  part  of  the  training 
department  rather  than  a separate  entity. 

0 Assign  SMEs  who  are  current  instructors  in  the 
training  squadron. 

0 Make  SME  assignments  on  the  basis  of  merit  with 
the  clear  understanding  that  participation  in  the 
ISD  program  is  a promotion  which  will  allow  those 
assigned  to  have  a large  impact  on  the  future  of 
the  weapons  system  training  program. 

0 Select  a team  leader  who  is  responsive  to  ISD  re- 
quirements and  who  will  provide  strong  direction 
to  the  ISD  team  while  working  closely  with  con- 
tractor specialists. 

0 Assign  SMEs  full  time  with  extra  duties  kept  to  a 
minimum. 

0 Select  SMEs  who  are  expected  to  remain  at  the  duty 
station  for  the  duration  of  the  ISD  program. 

0 Assign  a full  complement  of  SMEs  based  on  require- 
ments from  previous  similar  ISD  programs  and 
manpower  projections  for  the  current  program. 

Relative  to  other  ISD  programs,  overall  SME  support  for  the 
F-4  program  was  good.  The  ISD  department  was  created  and  staffed 
with  four  Marine  SMEs  prior  to  contract  award.  Initial  selection 
and  organization  of  SMEs  was,  therefore,  not  required.  Despite 
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the  advantage  of  early  organization  there  were  some  problems  with 
SME  support.  These  problems  were  as  follows: 

1.  There  was  insufficient  participation  by  Navy  SMEs. 

Navy  SMEs  from  VF-121,  NAS,  Miramar,  were  assigned  on 
an  intermittent  basis  with  no  SME  participating  more 
than  45  days.  This  situation  caused  a lack  of  con- 
tinuity in  Navy  input  and  complicated  the  process  of 
identifying  Navy/Marine  differences. 

2.  Marine  SMEs  were  assigned  other  duties,  such  as  flights 
and  simulator  lessons.  This  significantly  reduced  the 
amount  of  time  devoted  to  ISD  work. 

3.  In  addition  to  time  lost  due  to  other  duties,  there 
were  inefficiencies  caused  by  a lack  of  motivation. 
Usually  this  was  a transient  problem.  During  syllabus 
development,  however,  this  became  a significant  problem 
which  ultimately  caused  the  program  to  fall  behind 
schedule. 

4.  The  ISD  department  was  not  part  of  the  FRS.  Although 
this  organizational  structure  did  not  significantly 
hamper  the  program,  it  did  effect  communication  between 
the  SMEs  and  FRS  training  personnel,  i.e.,  members  of 
the  training  squadron  did  not  as  freely  interact  with 
the  SMEs  as  was  desired.  It  appeared  that  the  ISD 
SMEs  were  considered  "outsiders"  by  the  instructors. 

5.  The  number  of  SMEs  assigned  (4)  was  less  than  that 
specified  in  the  F-4  master  plan  (6  Marine,  2 Navy). 
Although  the  four  SMEs  assigned,  supplemented  by  Navy 
SMEs  as  discussed  above,  generally  performed  well, 
additional  SMEs  would  have  improved  program  progress 
and  quality. 

ISD  PROGRAM  ACCEPTANCE.  The  third  major  issue  to  be  discussed  is 
the  readiness  of  the  Navy  training  communities  to  accept  training 
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programs  produced  through  the  use  of  ISD.  This  issue  has  two 
facets:  (1)  the  reluctance  to  change  the  way  "it  has  always  been 
done-"  This  is  particularly  true  for  a weapons  sytem,  such  as  the 
F-4,  which  has  been  in  the  inventory  for  many  years  and  for  which 
training  doctrine  has  been  precisely  defined;  (2)  the  tendency 
for  the  ISD  program  to  degrade  over  time  with  a return  to  earlier 
training  techniques. 

To  help  insure  that  ISD  training  programs  will  be  accepted 
and  implemented  as  developed,  the  ISD  procedures  must  be  carried 
out  within  the  framework  of  existing  training  policy,  i.e.,  working 
through  the  ISD  procedures  per  se,  does  not  insure  viability 
of  the  end  product.  It  must  be  remembered  that  the  inertia  of  any 
well-established  system  works  against  changes.  Change  must, 
therefore,  be  closely  worked  out  with  those  who  perpetuate  the 
inertia.  In  the  case  of  an  aircrew  training  system,  currerit 
training  personnel,  as  well  as  policies  from  higher  authority 
are  the  primary  sources  of  inertia. 

In  applying  ISD  technology  to  an  existing  weapons  sytem,  ISD 
specialists  must  work  closely  with  training  personnel.  In  doing 
so,  the  development  process  becomes  one  of  "give  and  take."  Some 
aspects  of  the  emerging  training  program,  which  are  formulated 
using  ISD  procedures,  will  be  unacceptable  to  the  training  community 
or  will  counter  existing  rigid  policies.  These  aspects  must  be 
modified  to  achieve  acceptance.  On  the  other  hand,  existing 
training  procedures  may  be  better  than  or  enhance  training  pro- 
cedures resulting  from  an  ISD  analysis.  In  this  case,  the  outputs 
of  the  ISD  analysis  will  be  upgraded.  Although  intangible  and 
not  definable  in  concrete  procedural  steps,  this  high  degree  of 
interaction  during  training  program  development  is  a primary 
key  to  ultimate  success. 
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Once  an  ISO  program  has  been  implemented,  the  ISO  effort 
has  not  ended.  On  the  contrary,  a new  phase  of  ISO  has  begun. 
During  the  remainder  of  the  life  cycle  of  the  weapons  system, 
the  ISO-based  training  program  must  be  changed  and  updated  as 
the  weapons  system  changes  and  as  data  on  the  effectiveness  of 
the  training  program  is  accumulated.  To  do  this  a nucleus  of 
training  personnel  schooled  in  ISD  procedures  and  knowledge  in 
the  training  program  must  be  maintained.  In  addition,  resources 
must  be  provided  to  produce  and  implement  changes  as  they  are 
identified.  Without  a continued  emphasis  on  ISO,  even  the  best- 
formulated  training  program  will  fall  into  disarray. 

As  ISD  procedures  continue  to  be  refined,  a mechanism  is 
being  formulated  to  cope  with  the  acceptance  issue.  This  in- 
volves structuring  ISO  programs  so  that  all  relevant  parties  are 
included  from  program  inception  through  implementation  and  main- 
tenance. Relevant  parties  range  from  CNO  down  through  the  user 
squadrons  and  include  high  level  commanders,  government  educational 
specialists,  facilities  and  equipment  specialists,  contracting 
agencies,  contractors,  FRS  instructors,  and  procurement  personnel. 
This  structure  will  provide  command  emphasis  to  ISD  programs 
and  will  assist  in  coordinating  the  activities  of  the  many  agencies 
involved. 

General  acceptance  of  ISD  will  be  a gradual  process.  The 
organizational  concept  discussed  above  will  facilitate  the  process. 
The  biggest  boost  for  ISD,  however,  will  be  a history  of  successful 
programs . 

ISD  RESOURCES 

The  main  resource  used  in  this  project  was  labor.  Labor 
expended  by  category  and  task  is  summarized  in  Section  V,  Resource 
Utilization.  These  manhour  figures  are  accurate  as  will  be 
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discussed  below,  however,  the  figures  for  some  of  the  tasks  do 
not  provide  good  guidelines  for  future  planning  purposes. 

Since  the  development  of  objectives  hierarchies  and  SBOs 
began  prior  to  contract  award,  manhour  expenditures  for  this 
task  do  not  indicate  the  labor  required  to  perform  the  entire 
task.  Records  maintained  by  the  previous  contractor  for  the 
initial  portion  of  objectives  hierarchy  and  SBO  development  were 
• incomplete  and  were  not  included. 

About  50  percent  of  the  hierarchies  were  developed  under  Allen 
Corporation  direction.  Existing  hierarchies  were  reviewed  by 
the  SMEs  and  Allen  personnel  and  in  many  cases  were  revised.  It 
is  estimated  that  the  total  SME  time  required  to  learn  the  pro- 
cedures and  to  produce  the  hierarchies  and  SBOs  was  600  manhours. 

An  additional  100-200  professional  manhours  was  expended  by  the 
previous  contractor  in  the  early  stages  of  hierarchy  develdpment. 

Manhours  required  for  media  selection  are  good  estimates  for 
future  planning,  given  use  of  the  same  selection  model.  The 
revised  6.2  Specification  allows  latitude  in  the  choice  of  a media 
selection  model.  In  future  ISD  programs,  therefore,  other  models 
may  be  used  which  require  more  or  less  effort  than  the  one  used 
in  this  program. 

SME  support  was  lacking  during  syllabus  development.  This 
caused  more  calendar  time  than  anticipated  to  be  spent  on  this 
task.  The  increased  calendar  time  led  to  the  expenditures  of 
more  contractor  professional  manhours  than  would  have  been  re- 
quired if  SME  support  had  been  ample. 

. In  a previous  section,  several  suggestions  for  streamlining 

the  model  were  presented.  In  doing  so,  certain  efficiencies  would 
be  gained  in  implementing  ISD;  however,  it  is  difficult  to 
estimate  the  amount  which  would  be  gained.  The  labor  expendi- 
tures for  the  four  primary  contract  tasks,  plus  project  administration 
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and  reporting,  are  considered  to  be  about  the  minimum  for  an  ISD 
program  of  this  size.  It  is  projected  that  the  increased 
efficiency  resulting  from  streamlining  the  model  would  increase 
the  quality  of  the  products  rather  than  significantly  reduce 
manpower  requirements. 

Another  significant  resource  was  that  required  to  prepare 
and  duplicate  the  many  deliverable  end-products  specified  in  the 
contract.  Clearly,  monthly  progress  reports  and  the  final  report 
draft  and  final  versions  are  necessary  to  enable  the  contract 
technical  monitor  to  track  progress  and  to  get  an  overview  of 
project  activities.  It  is  recommended  that  requirements  for  the 
submittal  of  the  other  major  end  products  (Hierarchy  Report, 

Mediq  Report,  Syllabus  Report,  and  TSRA)  be  modified  in  future 
contracts.  These  recommendations  are  as  follows: 

1.  Delete  the  Media  Report  and  provide  a summary  of  media 
selections  in  the  Syllabus  Report. 

2.  Require  final  versions  only  rather  than  draft  and  final 
versions  of  each  of  the  four  documents. 

3.  If  the  Phase  I contract  includes  lesson  specifications, 
make  the  Syllabus  Report  a working  document  subject 

to  review  by  the  contract  technical  monitor  with  the 
syllabus  included  in  the  Lesson  Specification  Report. 

This  will  eliminate  the  need  for  Syllabus  Update 
Reports  during  lesson  specification  development. 

Since  the  present  contract  did  not  include  lesson 
specifications,  this  recommendation  is  not  pertinent 
to  F-4  ISD.  In  other  ISD  programs,  however.  Phase 
I has  included  lesson  specifications. 

The  net  result  of  these  recommendations  for  Phase  I activities 
is  that  one  version  of  the  Hierarchy  Report,  Syllabus  Report,  and 
TSRA  will  be  required.  The  Hierarchy  Report  contains  the  data 
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base  for  the  remainder  of  training  program  development  and  should 
be  submitted  for  review,  as  well  as  being  provided  in  multiple 
working  copies  for  use  in  subsequent  phases  of  the  program.  The 
Syllabus  Report  and  TSRA  are  the  final  products  prior  to  lesson 
specifications.  If  Phase  I terminates  with  these  products,  rather 
than  continuing  through  lesson  specifications,  both  are  required 
as  wrap-ups  to  Phase  I activities. 

DOCUMENTATION  ADJUSTMENTS 

In  the  discussion  of  ISO  Methods  in  this  section  reconimendations 
for  changes  to  the  ISO  specification  were  presented.  These 
recommendations  are  summarized  below: 

1.  In  the  discussion  of  procedures  used  in  hierarchy 
development  provide  better  guidelines  for  determining 
how  deep  a hierarchy  should  be.  It  is  suggested  that 
in  addition  to  the  current  guidelines  the  lowest 
appropriate  level  in  a hierarchy  be  defined  in  terms 
of  the  amount  of  instruction  required  to  teach  a base 
level  objective  in  ground  school. 

2.  Delete  conditions  and  standards  as  requirements  for  SBOs. 

3.  Augment  instructions  for  hierarchies  and  SBOs  so  that 
when  developed  the  hierarchies  and  SBOs  contain  systems 
and  training-specific  objectives,  as  well  as,  mission 
objectives.  The  systems  and  training-specific  objectives 
can  be  included  within  the  framework  of  the  mission 
objectives  without  altering  the  mission  orientation. 

4.  Incorporate  media  selection  into  syllabus  development 
and  select  media  by  lesson  rather  than  by  objective. 

This  would  eliminate  the  need  for  a separate  media 
report  and  may  require  reformulation  of  the  selection 
model . 
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5.  Restate  syllabus  guidelines  as  follows: 

a.  Base  the  number  of  flights  on  the  number  of  flight 
objectives  and  amount  of  practice  required  for 
each  rather  than  specifically  specifying  the 
maximum  feasible  number  of  flights. 

b.  Sequence  flights  from  "simple  to  more  complex" 
rather  than  from  "easier  to  more  difficult-" 

c.  Structure  units  on  the  basis  of  content  rather 
than  specifying  that  a flight  is  the  last  lesson 
of  any  unit  that  contains  a flight.  This  will 
allow  multiple  flights  containing  the  same  or  very 
similar  objectives  to  be  included  in  the  same  unit. 

6.  Expand  the  guidelines  for  the  TSRA  as  follows: 

a.  Identify  the  government  funding  sources  for  different 
costs.  This  will  help  contracting  personnel  determine 
which  costs  will  come  from  ISD  funds  and  which  will 
come  from  other  sources. 

b.  Further  define  equipment  and  facilities  categories 
in  detail  comparable  to  that  provided  for  materials 
production. 

c.  Include  costs  for  technical  work  and  documentation 
required  under  the  6.2  specification,  such  as 
implementation  and  quality  control  plans  and  an 
instructor  training  course. 

d.  Consider  revising  the  multiplication  factors  used 
to  estimate  revision  costs. 

e.  Break  revision  costs  down  so  that  costs  for  evalu- 
ation and  revision  specifications  are  distinct 

from  costs  for  materials  production  during  revision. 

f.  Sum  manhour  requirements  to  derive  the  number  of 
personnel  required  in  each  labor  category. 

7.  Increase  the  detail  in  the  DID  for  the  final  report,  so  that 
the  general  scope  of  each  section  is  well-defined.  Currently,  many 
of  the  section  headings  do  not  clearly  indicate  the  general  content 
areas  expected  to  be  covered.  Another  approach  is  to  make  the  DID 
more  general,  thus  allowing  contractors  greater  flexibility  in 
formulating  the  report. 
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A great  deal  was  accomplished  during  the  first  phase  of  the 
F-4  ISD  program.  A detailed,  prescriptive  ISD  model  was  studied, 
interpreted,  and  implemented  by  a contractor  not  involved  ih  the 
formulation  of  the  model.  Although  certain  deficiencies  were 
identified  as  a result  of  experience  during  implementation,  the 
model  essentially  stood  on  its  own  and  provided  definitive 
guidance  through  the  sequence  of  ISD  tasks.  The  viability  of 
the  model  was  not  challenged.  Recommendations  for  upgrading  the 
model,  as  well  as,  more  general  ISD  considerations  were  presented 
in  previous  sections.  Several  additional  points  will  be  discussed 
below. 

A large  data  base  was  collected  and  manipulated  in  this  pro- 
gram. At  times  there  were  information  management  problems  due 
to  the  size  of  the  data  base.  A possible  solution  to  this  problem 
is  the  use  of  a computerized  system  which  contains  algorithms  for 
production  of  the  data  base  (i.e.,  guides  SMEs  through  hierarchy 
and  SBO  development)  and  for  manipulation  of  the  data  base  (e.g., 
assists  in  organizing  and  sequencing  objectives  into  a syllabus). 

It  is  understood  that  computer  routines  for  this  purpose  have 
been  or  are  currently  being  developed.  The  application  of  these 
routines  to  aircrew  ISD  programs  should  be  studied. 

In  this  report  and  reports  on  other  ISD  programs,  considerable 
discussion  has  been  devoted  to  the  quality,  source,  and  number  of 
SMEs  participating  in  ISD  programs.  Very  little  discussion, 
however,  has  been  devoted  to  the  composition  of  the  contractor  ISD 
team.  Particularly  relevant  are  the  qualifications  of  the 
professional  personnel  who  work  closely  with  the  SMEs  in  producing 
the  various  ISD  end-products. 

The  guidelines  provided  by  NAVTRAEQUIPCEN  for  the  selection 
of  personnel  emphasize  educational  and  training  application. 
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Clearly  these  are  important  criteria,  particularly  for  the  program 
manager.  To  supplement  the  more  theoretical  training  personnel  an 
invaluable  asset  to  the  contractor  team  is  a person(s)  knowledge- 
able In  military  training  doctrine  and  in  the  weapons  system  under 
study  or  a similar  system  (i.e.,  a retired  or  discharged  SME).  This 
type  qf  person  is  able  to  relate  to  the  military  SMEs  and  to  solicit 
input  from  the  SMEs  that  otherwise  may  be  overlooked.  The  ability 
of  contractor  personnel  to  relate  to  military  personnel  is  an  in- 
tangible, yet  very  important,  component  of  a successful  implemen- 
tation of  1SD  procedures.  In  addition,  he  is  able  to  review  subject 
matter  content  for  completeness  and  accuracy,  thus  complimenting 
other  reviews  for  adherence  to  1SD  specifications. 

One  final  broad  consideration  is  the  readiness  of  the  Navy 
training  communities  to  accept  ISO-based  training  programs.  At  the 
present  they  are  not  ready  to  accept  ISD  in  its  purest  form.  The 
question  then  arises  whether  ISD  needs  to  change  or  the  training 
communities  need  to  change.  The  answer  is  that  both  need  to  adjust, 
ISD  to  military  doctrine  which  has  evolved  over  time  and  much  of 
which  is  sound,  and  the  training  communities  to  the  benefits 
inherent  in  a thorough,  systematic  training  analysis.  Current 
training  can  and  is  being  improved  through  the  application  of  ISD, 
but  only  to  the  degree  that  ISD  is  responsive  to  the  needs  of  the 
training  communities.  Increased  acceptance  of  ISD  will  come  as 
in-seryice  ISD  capabilities  are  developed  and  as  the  structure 
for  administration  and  control  of  ISD  programs  is  refined. 
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SECTION  V 

RESOURCE  UTILIZATION 

This  section  presents  an  account  of  the  resources  used  during 
the  F-4J/N  ISD  program.  There  are  separate  categories  for  personnel, 
time,  and  facilities  and  equipment.  The  purpose  for  collecting 
and  summarizing  resource  data  is  to  expand  upon  like  data  collected 
by  NAVTRAEQUIPCEN  from  other  ISD  programs.  This  data  can  be  used 
to  establish  a reliable  basis  for  planning  in  future  ISD  applications. 
More  specifically,  the  data  contained  herein  provides  resource 
requirements  for  implementation  of  the  6.2  ISD  model.  Since  this 
model  is  the  one  currently  being  refined  as  a standard  for  future 
Navy  ISD  programs,  resource  requirements  from  the  F-4  program  are 
particularly  relevant  for  future  planning. 

PERSONNEL 

Principal  on-site  technical  contributors  to  the  F-4  ISD  program 
were  an  Instructional  Psychologist  (IP),  who  provided  instruction 
and  guidance  in  ISD  techniques,  a Production  Manager  (PM),  who  co- 
ordinated ISD  and  administrative  activities,  and  Subject  Matter 
Experts  (SMEs).  Four  SMEs  were  initially  assigned  to  the  program. 

The  number  of  SMEs  actually  workiny  on  the  program  varied  throughout, 
ranging  from  three  to  six.  The  number  of  manhours  expended  for  each 
personnel  category  by  ISD  step  is  shown  in  Table  6. 


TABLE  6.  MANHOURS  EXPENDED  BY  ON-SITE 
ISD  TECHNICAL  PERSONNEL 


ISD  STEP 

if. 

PM 

SMEs 

Hierarchies  and  SBOs 

124 

250 

200 

Media  Selection 

115 

85 

42 

Syllabus 

385 

300 

401 

Training  Support 

75 

15 

0 

Requirements  Analysis 

... 



TOTAL 

699 

650 

643 
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The  data  presented  in  Table  6 reflect  actual  time  spent  on  each 
task.  Other  tasks,  such  as  project  administration,  preparation  of 
progress  reports  and  memos,  etc.,  are  not  shown. 

At  the  time  of  contract  award  to  Allen  Corporation,  the 
development  of  objectives  hierarchies  and  SBOs  was  approximately 
50  percent  complete.  The  manhour  totals  for  this  task  in  Table  6 are, 
therefore,  estimated  to  be  about  half  those  required  to  develop 
the  full  set  of  hierarchies  and  SBOs  for  the  F-4. 

The  total  on-site  contractor  personnel  hours  are  shown  in 
Table  7.  These  figures  contain  the  totals  from  Table  6 plus 
manhours  for  secretarial  and  administrative  functions. 

TABLE  7.  TOTAL  PROJECT-RELATED  MANHOURS 

SPENT  BY  ON-SITE  CONTRACTOR  PERSONNEL 


STAFF  MANHOURS 

Instructional  Psychologist  904 

Production  Manager  960 

Administrative  Assistant/Secretary  1,063 


Total  hours  of  off-site  personnel  by  category  are  shown  in 
Table  8.  The  categories  are  technical,  management,  and  support. 
Technical  hours  include  assistance  with  and  review  of  on-site 
activities  and  preparation  of  reports.  Management  hours  include 
contract  management,  organization  and  review  of  contract  deliver- 
ables, and  liaison  with  the  contract  technical  monitor.  Support 
hours  include  typing  and  general  secretarial  assistance  in  support 
of  off-site  technical  and  management  efforts. 


PERSONNEL  CATEGORY  MANHOURS 

Technical  440 

lianagement  575 

Support  185 
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The  overall  manhour  expenditures  were  in  accord  with  those 
originally  projected;  however,  individual  task  requirements 
differed  from  the  original  projections.  Since  the  development  of 
objectives  hierarchies  and  SBOs  began  prior  to  contract  award, 
completion  of  this  task  required  fewer  manhours  than  projected. 
Media  selection  also  required  less  time  than  projected.  Syllabus 
development  was  a difficult  task.  The  difficulty  of  the  task 
combined  with  a shortage  of  SME  support  during  this  portion  of 
the  contract  caused  more  contractor  hours  to  be  used.  As  a result, 
the  milestone  for  syllabus  completion  was  not  met.  This  will  be 
discussed  further  in  the  next  section.  Finally,  the  Training 
Support  Requirements  Analysis  was  completed  as  projected. 


\ | 


A summary  of  the  projected  versus  actual  elapsed  time  for  each 
ISD  step  is  shown  in  Figure  11.  The  program  was  on  or  ahead  of 
schedule  through  media  selection.  As  discussed  in  the  "Problems 
and  Solutions"  section  under  "Syllabus  Development,"  problems  were 
encountered  which  precluded  the  completion  of  syllabus  development 
as  scheduled.  These  problems  were  the  brief  amount  of  time  allotted 
for  syllabus  development,  the  inherent  difficulty  of  the  task,  the 
lack  of  agreement  among  SMEs,  a lack  of  adequate  SME  support,  and 
the  multiple  syllabi  which  had  to  be  developed.  Failure  to  complete 
the  syllabi  on  or  near  schedule  caused  the  Training  Support  Re- 
quirements Analysis  (TSRA)  and  the  Final  Report  to  be  delayed. 

At  the  outset  it  was  recognized  that  the  schedule  requirements 
for  this  program  were  demanding.  Times  allotted  to  the  four  major 
tasks  were:  hierarchies  and  SSOs  (including  SME  training)  - 11  weeks; 
media  selection  - 4 weeks;  syllabus  development  - 4 weeks;  and 
TSRA  - 4 weeks.  Four  weeks  each  for  media  selection  and  the 
training  support  requirements  analysis  were  sufficient  to  complete 
the  tasks.  As  discussed  above,  the  four  weeks  scheduled  for 
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syllabus  development  was  not  sufficient  for  a program  of  this 
size  in  which  multiple  syllabi  are  required.  It  is  recommended 
that  in  similar  future  ISD  programs,  a minimum  of  ten  weeks  be 
scheduled  for  this  task.  With  adequate  SME  support,  it  is 
hypothesized  the  syllabi  required  in  the  F-4  ISO  program  could 
have  been  completed  in  ten  weeks. 

The  objectives  hierarchies  and  SBOs  were  completed  on 
schedule;  however,  this  was  partly  a function  of  their  having  been 
started  prior  to  contract  award.  For  future  planning  purposes, 
it  is  recommended  that  in  programs  of  this  size,  in  which  SMEs 
are  the  primary  developers,  a minimum  of  15  weeks  be  allowed  for 
SME  training  and  the  development  of  objectives  hierarchies  and 
SBOs. 

FACILITIES  AND  EQUIPMENT 

All  facilities  and  equipment  were  government  furnished  and 
were  adequate  for  the  program.  The  four  assigned  SMEs  from 
MCCTRG-10  occupied  their  normal  workspaces  in  two  offices.  A 
large  conference  room  was  available  for  use  by  SMEs  who  supplemented 
the  four  permanently  assigned  and  for  group  meetings,  training 
sessions,  etc.  Two  offices  with  desks  were  provided  for  contractor 
personnel.  One  was  used  by  technical  personnel.  The  other  was 
used  by  the  administrative  assistant/secretary  and  also  contained 
filing  cabinets  and  a work  table.  All  rooms  adjoined  thus  enhancing 
interaction  among  all  personnel  involved  in  the  program. 

Reproduction  capability  was  marginal  when  contract  work  began. 
During  the  contract,  however,  this  capability  was  upgraded  by  the 
government  and  was  quite  satisfactory  for  the  remainder  of  the 
contract.  Government  furnished  typewriters  were  IBM  Mag  Card  I Is , 
which  significantly  facilitated  production  of  the  numerous  large 
documents  required  under  the  contract. 

It  should  be  noted  that  the  facilities  and  equipment  were  a 
very  positive  factor  for  both  the  SMEs  and  contractor  personnel. 
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They  provided  a good  working  environment  both  in  terms  of  space 
and  capabilities. 

The  contractor  provided  consumable  supplies  and  a telephone 
on-site.  All  off-site  activities  took  place  in  contractor 
facilities.  Since  end  products  of  the  four  major  tasks  were  pro- 
duced on-site,  off-site  activities  primarily  involved  review, 
accounting,  and  management  functions  for  which  few  resources  were 
requ i red . 
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APPENDIX  A 


STRUCTURE  OF  OBJECTIVES  HIERARCHIES 


This  appendix  is  an  extract  with  minor  modifications 
from  one  section  of  the  Objectives  Hierarchy  Report 
(CDRL  Item  E005).  It  presents  a description  of  the  organi- 
zation of  the  hierarchies,  the  numbering  system,  and  the 
symbols  used. 
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STRUCTURE  OF  OBJECTIVES  HIERARCHIES 
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The  hierarchies  and  SBO's  were  structured  around  the 
four  F-4  missions:  (1)  Air  to  Surface,  (2)  Air  to  Air, 

(3)  Reconnaissance,  and  (4)  Escort.  Each  of  the  four  missions 
was  partitioned  into  the  same  ten  mission  segments: 

1.  Premission  Phase 

2.  Brief  Phase 

3.  Prelaunch  Phase 

4.  Takeoff/Departure  Phase 

5.  Navigation  Phase 

6.  Tactical  Phase 

7.  Refueling  Phase 

8.  Approach/Landing  Phase 

9.  Post  Mission  Phase 

10.  Debrief  Phase 

It  was  from  this  basic  structure  of  four  missions  by  ten  mission 
segments  that  the  hierarchies  and  SBO's  were  developed. 

The  numbering  of  the  SBO's  indicates  the  hierarchical 
relationships  among  the  SBO's  which  inake  up  the  hierarchy  dia- 
grams. The  first  number  specifies  one  of  the  four  missions  cited 
above  while  the  second  number  indicates  one  of  the  ten  mission 
segments  within  the  mission,  e.g.,  1.1  is  mission  one  (Air  to 
Surface),  mission  segment  one  (Premission  Phase).  The  subsequent 
digits  in  the  SBO  numbers  are  derived  from  the  structures  of  the 
hierarchies  and  relate  the  prerequisite,  lower-level  component 
skills  and  knowledge  to  the  higher  level  skills  and  knowledge. 

The  hierarchies  are  numbered  from  top  to  bottom  and  within 
a branch  at  the  same  level  from  left  to  right.  Each  subordinate 
objective  in  a hierarchy  has  a reference  number  which  includes 
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the  complete  reference  number  from  its  superordinate  objective 
in  the  hierarchy  and  an  added  entry  (Figure  A-l).  If  multiple 
objectives  are  immediately  subordinate  to  the  same  objective, 
the  reference  numbers  for  all  of  them  will  be  identical  in  all 
but  the  last  entry.  The  final  entries  for  the  subordinate 
objectives  are  sequentially  assigned  numbers  beginning  with  1. 

In  Figure  A-l  the  first  hierarchy  for  Mission  Segment  1.1 
is  shown.  In  order  to  conduct  premission  planning  the  pilot  must 
"Gather  Data  for  Mission"  (1.1.1),  "Select  Pubs  for  Gathering 
Mission  Data"  (1.1.2),  "Compute  Air  to  Surface  Mission  Data" 
(1.1.3),  "Record  Data  for  Mission"  (1.1.4),  and  "Evaluate  Data 
on  Mission  Planning"  (1.1.5). 

Three  different  types  of  blocks  are  shown  in  Figure  A-l. 

The  dashed  rectangular  block  at  the  top  of  the  page  (1.0)  in- 
dicates that  the  page  is  a continuation  of  another  hierarchy 
page.  The  objective  within  a dashed  block  is  that  objective 
which  is  immediately  superordinate  to  the  objectives  on  the 
current  hierarchy  page. 

The  first  solid  rectangular  block  following  a dashed 
rectangular  block  (1.1  in  Figure  A-l)  is  always  an  objective 
which  has  been  carried  over  from  a previous  page,  i.e.,  there  was 
insufficient  room  on  a previous  page  to  adequately  analyze  the 
objective,  therefore,  it  was  continued  on  a subsequent  page. 

Other  terminal  solid  rectangular  blocks  on  a page  are  not  analyzed 
further  on  a subsequent  page.  (1.1. 4.1  and  1. 1.4.2  in  Figure  A-l). 

The  "hex"  box  is  used  to  indicate  an  objective  which  is 
further  analyzed  on  a succeeding  page  (1.1.1,  1.1.2,  1.1.3,  and 
1.1.5  in  Figure  A-l).  The  objective  in  a "hex"  box  is  contained 
in  the  top  solid  rectangle  on  a subsequent  hierarchy  page.  The 
superordinate  objective  to  it  is  included  in  the  dashed  rectangle 
at  the  top  of  that  page. 

Figure  A-2  is  a further  analysis  of  the  hex  box  1.1.2  in 
Figure  A-l.  Note  that  the  immediately  superordinate  objective  1.1 


i 


1 


101 


N AVTRAEQU I PCEN  77-C-0081-1 


iple  Hierarchy:  Continuation  Hierarchy 
i Hex  Box  1.1.2  In  Figure  A-1 


NAVTRAEQU I PCEN  77-C-0081-1 


is  contained  in  the  dashed  rectangle  at  the  top  of  the  page 
and  that  1.1.2  which  was  a hex  box  in  Figure  A-l  is  the  first 
solid  rectangular  box  following  the  dashed  rectangular  box. 

In  Figure  A-2  all  objectives  are  contained  in  solid  rectangular 
boxes  thus  indicating  that  they  will  not  be  analyzed  further  on 
succeeding  pages. 

One  or  more  SBO  pages  accompany  each  hierarchy.  The  SBO 
listing  contains  all  information  (i.e.,  number,  complete  name, 
conditions,  and  standards)  for  all  objectives  in  solid  rectan- 
gular boxes  in  the  accompanying  hierarchy.  For  objectives  in 
"hex"  boxes,  which  will  be  analyzed  further  on  another  page, 
only  the  number  and  objective  name  is  present  on  the  SBO  page. 
Figure  A-3  is  the  accompanying  SBO  page  for  the  hierarchy  shown 
in  Figure  A-l;  Figure  A-4  accompanies  the  hierarchy  in  Figure 
A-2. 

If  the  same  objective  appears  on  more  than  one  objective 
page  it  has  alternate  reference  numbers.  The  duplication  of 
an  objective  on  different  hierarchy  pages  is  indicated  in  two 
ways.  First,  such  an  objective  contains  a small  solid  triangle 
in  the  upper  left  corner  of  the  reference  number  section  of  the 
box  (Figure  A-2).  Secondly,  all  objectives  with  an  alternate 
reference  mark  are  listed  in  an  alternate  numbers  table.  There 
is  one  such  table  at  the  end  of  the  pilot  hierarchies  and  one 
at  the  end  of  the  RIO  hierarchies. 

As  the  hierarchies  were  being  developed  clear  patterns 
emerged  within  mission  segments,  across  missions.  In  many  cases 
the  hierarchies  and  SBO's  comprising  a segment  within  one  mission 
were  identical  to  those  in  the  same  mission  segment  within  one 
or  more  other  missions.  For  example,  the  hierarchies  in  the 
Prelaunch  segment  were  identical  across  all  four  missions  for 
both  the  pilots  and  RIO's.  Within  both  the  pilot  and  RIO  mission 
segments,  17  of  the  40  mission  segments  were  unique  with  the 
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remaining  23  segments  being  duplicates  of  the  17  unique  segments. 

Due  to  the  high  degree  of  redundancy  across  mission  segments, 
the  pilot  and  RIO  hierarchies  contained  only  those  which  were 
unique.  For  both  the  pilot  and  RIO  hierarchies  Mission  1.0  (Air 
to  Surface)  is  presented  in  its  entirety,  i.e.,  all  ten  mission 
segments  are  included.  For  the  remaining  three  missions  2.0 
(Air  to  Air),  3.0  (Reconnaissance),  and  4.0  (Escort)  only  those 
mission  segments  whose  hierarchies  differ  from  those  in  the 
corresponding  mission  segment  in  Mission  1.0  were  presented. 
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APPENDIX  B 


MEDIA  DEFINITIONS  AND  SAMPLE  SELECTION  DIAGRAMS 


This  appendix  contains  the  media  for  each  of  the 
terminal  blocks  in  the  selection  diagram,  definitions 
of  each  of  the  candidate  media,  and  examples  of  the 
selection  process. 
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CANDIDATE  MEDIA 


Each  path  in  the  diagram  shown  in  Figure  8 leads  to 
recommended  media  (labeled  Ml  through  M44)  for  presenting 
the  behavioral  objective.  These  recommended  media  are  shown 
in  Figure  B-l.  In  most  cases  primary  and  alternate  media 
are  recommended.  However,  in  some  cases  one  or  more  possible 
answers  to  a question  are  inappropriate  for  any  objective 
that  has  entered  a certain  pathway.  In  these  cases  (such 
as  Media  #1)  the  terminal  blocks  indicate  that  the  combin- 
ation of  answers  to  questions  violated  a principle  of  media 
selection.  The  remedial  solution  is  to  review  the  path  and 
correct  the  inconsistency. 

MEDIA  DEFINITIONS. 

The  instructional  media  contained  in  the  model  are: 

0 Mediated  Interactive  Lecture 
0 Workbook 

0 Slide  Tape  Presentation 
0 Random  Access  Slide  Presentation 
0 Videotape  Presentation 
0 Computer  Assisted  Instruction 
0 Trainer  Exercise 
0 Aircraft  Flight 
Each  of  the  media  is  defined  below. 

Mediated  Interactive  Lecture  (MIL)  - The  major  portion  of 
the  instructional  material  in  an  MIL  is  presented  verbally  by  an 
instructor  to  a group  of  students.  A MIL  requires  two  hard  copy 
products,  student  worksheets  and  visual  aids.  The  student  worksheet 
includes  a set  of  lesson  objectives,  the  generalities  for  each 
objective,  necessary  charts,  tables,  and  figures,  and  a set  of 
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Media  # 1 

Familiarization  level  content  should  not  be  taught 
at  a recall  level . 

Media  if 2 

Familiarization  level  content  should  not  be  taught 
a rule-using  level . 

Media  H 3 

Paired  associate  level  _ontent  should  not  be  taught 
at  a rule-using  level. 

Media  #4 

First  Choice:  Workbook 

Second  Choice:  Mediated  interactive  lecture 

Media  #5 

First  Choice:  Mediated  interactive  lecture 
Second  Choice:  Slide-tape  presentation 

Media  #6 

First  Choice:  Videotape 

Warning:  It  may  not  be  worth  the  expense 

Media  #7 

It  is  probably  a waste  of  time  and  resources  to  teach 
this  objective  at  a familiarization  level. 

Media  #8 

First  Choice:  Slide  tape 

Second  Choice:  Mediated  interactive  lecture 

Media  09 

First  Choice:  Videotape 

Second  Choice:  Mediated  interactive  lecture 

Media  #10 

Why  is  an  interactive  presentation  needed  to  teach 
discriminated  recall  level  behavior? 


Figure  B-l.  The  Meaning  of  Each  of  the  44  Terminal 
Blocks  in  the  Media  Selection  Network 
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Med i a fill 

You  may  want  to  combine  this  objective  with  the 
classification  level  objective  dealing  with  this 
content. 

First  Choice:  Workbook 

Second  Choice:  Mediated  interactive  lecture 
Media  HZ 

First  Choice:  Videotape 

Second  Choice:  Mediated  interactive  lecture 

Media  #13 

Why  do  you  need  an  interactive  simulation  to  teach 
a discriminated  recall  level  behavior? 

Media  #14 

First  Choice:  A simulator  or  the  actual  equipment 
and  a worksheet. 

Second  Choice:  CAI  simulation 

MediaJMJj 

First  Choice:  Workbook 

Second  Choice:  Slide  tape 

Third  Choice:  Mediated  interactive  lecture 

Media  #16 

First  Choice:  CAI  (memory) 

Second  Choice:  Workbook 
Third  Choice:  Slide  tape 

Media  # 17 

First  Choice:  CAI  (memory) 

Second  Choice:  Slide  tape 
Third  Choice:  Slide  tape 

Media  #18 

First  Choice:  CAI  (memory) 

Second  Choice:  Slide  tape 
Third  Choice:  Workbook 

Media  #19 

First  Choice:  Videotape 

Second  Choice:  Mediated  interactive  lecture 


Figure  B-l.  The  Meaning  of  Each  of  the  44  Terminal 
Blocks  in  the  Media  Selection  Network 
(continued) 
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Media  #20 


Why  do  you  need  an  interactive  presentation  to  teach 
a discriminated  recall  level  behavior? 

Media  #21 

You  may  want  to  combine  this  objective  with  the  work- 
book portion  of  the  rule-using  level  objective  dealing 
with  the  content. 

First  Choice:  Workbook 

Second  Choice:  Mediated  interactive  lecture 

Media  #22 

First  Choice:  CA1  (memory) 

Second  Choice:  Workbook 
Third  Choice:  Slide  tape 

Media  #23 

You  may  want  to  combine  this  with  the  workbook  portion 
of  the  rule-using  level  objective  dealing  with  this 
content. 

First  Choice:  Slide  tape 
Second  Choice:  Workbook 

Media  #24 

First  Choice:  Videotape 

Second  Choice:  Mediated  interactive  lecture 

Media  #25 

First  Choice:  Videotape 

Second  Choice:  Mediated  interactive  lecture  (with  VT). 
Media  #26 

First  Choice:  Simulator 
Second  Choice:  CA1  simulation 
Third  Choice:  Videotape 

Media  #27 

First  Choice:  Videotape 

Second  Choice:  Mediated  interactive  lecture  (with  VT). 
NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective  at 
the  discriminated  recall  level. 


Figure  B-l . The  Meaning  of  Each  of  the  44  Terminal 
Blocks  in  the  Media  Selection  Network 

(continued) 
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Media  #28 

First  Choice: 
Second  Choice: 
Third  Choice: 

Media  #29 

First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 

Media  #30 

First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 

Media  #31 

First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 


Simulator 

CAI  Simulation 

Videotape 


Workbook 

CAI 

Mediated  interactive  lecture 
Random  access  slide-workbook 


CAI 

Random  access  slide-workbook 
Workbook 

Mediated  interactive  lecture 


Random  access  slide-workbook 
CAI 

Mediated  interactive  lecture 
Workbook 


Media  #32 


First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 


CAI 

Random  access  slide-workbook 
Mediated  interactive  lecture 
Workbook 


Media  #33 


First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 


Workbook 

CAI 

Mediated  interactive  lecture 
Random  access  slide-workbook 


Media  #34 


First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 
NOTE: 


CAI 


Random  access  slide-workbook 
Workbook 

Mediated  interactive  lecture 
Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective  at 
the  discriminated  recall  level. 


Figure  B-l.  The  Meaning  of  Each  of  the  44  Terminal 
Blocks  in  the  Media  Selection  Network 
(continued) 
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Media  #35 

First  Choice:  Random  access  slide-workbook 
Second  Choice:  CAI 

Third  Choice:  Mediated  interactive  lecture 
Fourth  Choice:  Workbook 

NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective 
at  the  discriminated  recall  level. 

Media  #36 

First  Choice:  CAI 

Second  Choice:  Random  access  slide-workbook 
Third  Choice:  Mediated  interactive  lecture 
Fourth  Choice:  Workbook 

NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective 
at  the  discriminated  recall  level. 

Media  #37 

First  Choice:  Workbook 
Second  Choice:  CAI 

Third  Choice:  Mediated  interactive  lecture 
Fourth  Choice:  Random  access  slide-workbook 

Media  #38 

First  Choice:  CAI 

Second  Choice:  Random  access  slide-workbook 

Third  Choice:  Workbook 

Fourth  Choice:  Mediated  interactive  lecture 

Media  #39 

First  Choice:  Workbook 
Second  Choice:  CAI 

Third  Choice:  Mediated  interactive  lecture 
Fourth  Choice:  Random  access  slide-workbook 
NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective 
at  the  discriminated  recall  level. 

Media  #40 

First  Choice:  CAI 

Second  Choice:  Random  access  slide-workbook 
Third  Choice:  Workbook 
Fourth  Choice:  Mediated  interactive  lecture 
NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective 
at  the  discriminated  recall  level. 


Figure  B-l.  The  Meaning  of  Each  of  the  44  Terminal 
Blocks  In  the  Media  Selection  Network 
(continued) 
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Media  #41 


First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 


Media  #42 


First  Choice: 
Second  Choice: 
Third  Choice: 
Fourth  Choice: 


Media  #43 


Random  access  slide-workbook 
CAI 

Mediated  interactive  lecture 
Workbook 


Random  access  slide-workbook 
Mediated  interactive  lecture 
Workbook 


First  Choice:  Random  access  slide-workbook 
Second  Choice:  CAI 

Third  Choice:  Mediated  interactive  lecture 
Fourth  Choice:  Workbook 

NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective 
at  the  discriminated  recall  level. 


Media  #44 


First  Choice:  CAI 

Second  Choice:  Random  access  slide-workbook 
Third  Choice:  Mediated  interactive  lecture 
Fourth  Choice:  Workbook 

NOTE:  Be  sure  you  have  a separate  objective  to  teach 
the  large  memory  component  of  this  objective 
at  the  discriminated  recall  level. 


Figure  B-l.  The  Meaning  of  Each  of  the  44  Terminal 
Blocks  In  the  Media  Selection  Network 
(continued) 


116 


NAVTRAEQU I PCEN  77-C-0081-1 


practice  item  response  sheets  for  each  objective.  Types  of  visual 
aids  used  in  a MIL  may  include  overhead  transparencies,  slides, 
or  videotapes,  depending  on  the  display  requirements  of  the  lesson. 
These  aids  are  used,  where  appropriate,  to  present  supporting 
information,  sets  of  examples,  and  sets  of  practice  items.  For  a 
lecture  to  be  adaptive  to  student  needs  it  must  be  interactive. 

An  interactive  lecture  provides  the  instructor  with  an  outline 
which  requires  him  to  ask  the  students  questions  and  alter  his 
presentation  based  on  student  responses. 

Workbook  (WB)  - The  defining  characteristic  of  a workbook  is 
that  all  instructional  components  are  presented  in  printed  form. 

The  workbook  begins  with  a lesson  map,  a lesson  introduction,  and 
a set  of  instructions  explaining  how  to  proceed.  Each  segment 
(corresponding  to  one  objective)  begins  by  presenting  the  objective, 
the  generality,  and  the  supporting  explanation  or  help.  The 
practice  items  are  designed  to  require  the  student  to  behave  at  the 
level  designated  in  the  objective.  The  practice  set  is  accompanied 
by  a feedback  and  help  section. 

Slide-Tape  Presentation  (ST)  - A slide-tape  presentation  shall 
contain  three  separate  components:  (1)  a set  of  slides  or  photographic 
frames,  (2)  audio  sections,  and  (3)  a student  worksheet.  The 
student  worksheet  first  presents  a lesson  map,  and  a set  of 
instructions  about  where  to  get  the  slides  and  the  audio  tape, 
where  and  how  to  set  them  up,  and  how  best  to  use  them.  For  each 
segment,  the  worksheet  presents  the  objective,  the  generality, 
necessary  charts  and  tables,  and  a response  form  for  the  practice 
items.  The  audio  tape  shall  begin  with  an  introduction.  For 
each  objective,  the  generality  is  restated  and  additional  supporting 
information  is  presented.  The  slides  shall  complement  the  audio 
tape.  A set  of  practice  items  is  presented  next.  The  student 
is  told  to  stop  the  tape  after  each  question,  and  write  his 
answer  on  his  worksheet.  When  the  tape  is  started  again,  it  gives 
feedback  and  help  for  that  item. 
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Random-Access  Slide  Presentation  (RAS)  - A random-access 
slide  presentation  shall  contain  two  components:  a set  of  slides 
presented  via  a random-access  slide  projector,  and  a student 
worksheet.  This  medium  allows  for  visual  presentation  of  large 
numbers  of  examples  and  practice  items  without  sacrificing  the 
freedom  of  movement  (i.e.,  learner  control)  lost  in  the  linear 
ST  or  VT  presentation.  The  worksheet  in  the  RAS  shall  present 
the  lesson  map,  a lesson  introduction,  and  instructions  for 
obtaining  the  slide  set,  setting  up,  and  viewing,  and  how  best  to 
use  the  lesson  materials.  For  each  segment,  it  presents  the 
objective,  the  generality  and  some  supporting  explanation.  It 
then  presents  tables  of  numbered  example  items  and  practice  items. 
Each  table  presents  a listing  of  the  slide  numbers  of  example, 
and  help,  or  practice  and  feedback  items.  The  student  can  use 
these  tables  to  check  those  examples  he  has  viewed,  or  the  practice 
items  he  has  worked.  The  practice  table  also  provides  space  for 
the  student's  response.  The  random-access  slides  are  used  to 
present  an  expanded  version  of  the  generality,  the  set  of  examples 
with  helps,  practice  items,  and  feedback. 

Videotape  Presentation  (VT)  - A videotape  presentation  shall 
contain  two  components:  a videotape  cassette  and  a student  worksheet. 
The  use  of  these  two  components  shall  be  parallel  to  the  use  of 
corresponding  components  of  the  slide-tape  presentation.  The  major 
difference  between  these  two  media  is  the  type  of  display  capability. 
The  instructional  strategy  used  is  similar  in  both  types  of  media. 

Computer  Assisted  Instruction  (CAI)  - In  this  medium  a 
computer  monitors  student  performance  through  frequent  tests  and 
optimizes  trainee  learning  by  branching  to  the  appropriate  level 
of  detail.  When  trainees  elicit  a strong  understanding  of  the 
content  the  computer  omits  further  instruction.  When  the  trainee 
fails  to  master  a subject  the  computer  branches  to  more  detailed 
instruction,  much  as  a human  instructor  does.  Due  to  the  high 
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cost  of  computer  assisted  instruction,  Allen  Corporation  was 
instructed  to  omit  CAI  from  consideration  on  this  training 
program. 

Trainer  Exercise  - A trainer  exercise  shall  be  used  to  allow 
the  student  to  practice  in  a simulated  real  world  environment. 

The  type  of  trainer  used  depends  on  whether  cockpit  motion  is 
required,  and  upon  the  degree  of  fidelity  to  the  real  world 
required.  A trainer  exercise  shall  also  include  three  other  com- 
ponents: (1)  a student  worksheet,  (2)  a student  evaluation  sheet, 
and  (3)  a trainer  feedback  sheet.  The  student  worksheet  presents 
the  student  with  the  set  of  lesson  objectives,  a lesson  intro- 
duction, and  a set  of  instructions  describing  what  he  should  do 
to  be  prepared  for  the  trainer.  The  worksheet  then  presents  any 
information  the  student  will  need  for  mission  planning  purposes, 
and  spells  out  exactly  what  planning  he  will  need  to  have  prepared. 
The  worksheet  then  outlines  in  detail  the  procedures  or  actions 
the  student  should  perform.  For  procedures  not  previously  practiced, 
the  worksheet  presents  a list  of  steps  to  perform.  Previously 
mastered  procedures  are  referred  to  by  name  without  the  accompanying 
list  of  steps.  In  all  cases,  the  worksheet  outlines  verbally  or 
pictorially  the  correct  result  of  the  action.  The  student  evaluation 
sheet  is  used  by  the  instructor  to  check  the  student’s  performance. 

It  describes  evaluation  instructions,  outlines  necessary  instructor- 
student  interactions,  and  gives  a checklist  of  all  points  that 
should  be  evaluated.  Based  on  the  student's  performance,  the 
instructor  fills  out  a trainer  feedback  sheet.  This  sheet  is 
formatted  so  the  instructor  can  check  areas  where  the  student 
needs  more  practice.  Each  area  references  course  lessons  which 
deal  with  the  area. 

Aircraft  Flight  (AC)  - The  instructional  materials  for  flight 
shall  be  similar  to  those  for  a trainer  exercise.  These  materials 
include  a student  worksheet,  a student  evaluation  form,  and  a flight 
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feedback  sheet.  The  student  worksheet  outlines  the  lesson  objectives, 
gives  the  student  necessary  data  and  instructions  for  mission 
planning,  and  describes  the  general  sequence  of  events  that  should 
occur  during  the  flight.  The  student  evaluation  sheet  and  the 
flight  feedback  sheet  are  the  same  format  as  those  used  in  the 
trainer  exercise.  The  student  evaluation  sheet  shall  be  printed 
on  knee-cards  so  it  can  be  carried  on  the  flight. 

SELECTION  PROCESS  EXAMPLES 

As  discussed  in  Section  III,  media  for  presenting  each  objective 
were  selected  by  asking  a series  of  five  questions.  Answers  to 
the  questions  traced  a path  through  the  media  selection  diagram. 

Figure  B-2  contains  a copy  of  the  media  selection  diagram  for 
pilot  behavioral  objective  1.3.4. 1.1  (State  Procedures  for  Taxi 
On  Field).  The  dark  lines  indicate  the  questions  asked  about  the 
objective  and  the  pathways  chosen  based  on  the  answers. 

The  algorithm  starts  at  the  triangle  in  the  center  of  the 
page  and  proceeds  to  Question  1 (What  is  the  level  of  behavior 
expected  of  the  student  in  this  segment?).  The  answer  is  Mo.  2 
(discriminated  recall)  because  the  student  is  required  to  recall 
and  state  the  procedures. 

The  algorithm  then  proceeds  to  Question  2 (What  level  of 
content  is  being  taught  in  this  segment?).  The  answer  is  No.  3 
(concept)  because  the  student  must  state  the  concept  or  pro- 
cedures but  not  manipulate  it  as  in  No.  4. 

Next  is  Question  5 (Is  the  memorization  component  of  this 
objective  large  or  small?).  The  answer  is  No.  2 (large)  because 
there  is  no  checklist  for  taxi  procedures  and  much  information 
must  be  remembered  to  correctly  perform  taxi  operations. 

The  final  question  is  No.  4 (What  is  the  minimum  display 
requirement?).  The  answer  is  No.  2 (verbal  and/or  symbolic  and/or 
static  complex  pictorial)  because  the  long  list  of  procedures 
will  require  complex  static  pictorial  presentation. 
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The  algorithm  has  led  to  Media  18.  As  shown  in  Figure  B-l, 
the  first  choice  for  Media  18  is  CAI.  Due  to  cost  considerations 
this  option  was  omitted.  Consequently,  the  recommended  media  for 
presentation  of  the  objective  (State  Procedures  for  Taxi  On  Field) 
are  slide  tape  (first  choice)  and  workbook  (second  choice). 

Figure  B-3  contains  a copy  of  the  media  selection  map  for 
pilot  behavioral  objective  1.3.4. 1 (Perform  Taxi  On  Field).  The 
first  question  concerns  the  level  of  behavior  required  of  the 
student.  The  answer  is  No.  3 (rule  using)  because  the  student 
must  apply  the  rule  in  actual  practice. 

Question  No.  2 asks  the  level  of  content  to  be  taught.  The 
answer  is  No.  4 (rule)  because  the  student  must  manipulate  the 
concept. 

The  last  question  (No.  4)  asks  what  are  the  minimum  display 
requirements.  The  answer  is  No.  4 (interactive)  because  the 
display  system  must  sense  student  inputs  and  feedback  actual 
aircraft  responses. 

The  algorithm  has  selected  Media  14.  The  first  choice  for 
Media  14  is  a simulator  or  the  actual  equipment  and  a worksheet. 
The  second  choice  is  CAI  and  was  eliminated. 
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APPENDIX  C 
ANALYSIS  EXAMPLE 


This  appendix  contains  an  example  which  traces  objectives 
from  the  task  listing,  through  hierarchy  and  SBO  development 
and  media  selection,  to  the  syllabus. 
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ANALYSIS  EXAMPLE 


In  the  sequence  of  Figure  C-l  through  C-8  on  the  following 
pages,  three  objectives  from  the  task  listing  are  traced  through 
three  analysis  steps.  First  they  are  broken  down  into  their 
prerequisite  skills  and  knowledges  in  an  objectives  hierarchy. 

Next,  media  are  selected  for  one  of  these  objectives  and  two  of 
the  prerequisite  objectives,  using  media  selection  diagrams. 

Finally,  the  objectives  from  the  hierarchy  are  grouped  into 
syllabus  lessons. 

The  task  listing  page  shown  in  Figure  C-I  was  the  start 
point  for  the  analysis.  Objectives  marked  with  asterisks  (*) 
are  those  which  are  analyzed  in  the  subsequent  hierarchy. 

Figure  C-2  is  the  hierarchy  containing  the  superordinate  objectives, 
the  objectives  of  interest  from  the  task  listing,  and  the  sub- 
ordinate objectives  identified  during  the  hierarchy  analysis. 

Note  that  objective  2. 6. 6. 4 (Determine  the  Optimum  Lesson  to  Use) 
is  at  the  same  level  in  the  hierarchy  as  the  three  objectives  from 
the  task  listing.  Although  the  inclusion  of  this  objective 
resulted  from  analysis  of  the  other  three  objectives  at  the  same 
level,  it  was  placed  as  a coordinate  objective  as  per  the 
specification  guidelines  for  hierarchy  development.  Figure  C-3 
is  the  SBO  pages  for  the  hierarchy  in  Figure  C-2. 

The  next  step  was  to  select  instructional  media  for  the 
objectives.  Figures  C-4,  C-5,  and  C-6  are  media  selection 
diagrams  for  three  of  the  objectives  in  Figure  C-2. 

Following  media  selection,  the  objectives  were  grouped  to 
form  lessons.  Two  lessons.,  one  ground  school  and  one  flight, 
were  formed  from  the  objectives  in  Figure  C-2.  The  syllabus 
sheets  for  these  lessons  are  shown  in  Figure  C-7  and  C-8. 
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%# 

F-4  AIRCREW  TASK  LISTING  VALIDATION 

FIlotI 


TASK  CONDITIONS  STANDARDS 


2. 6. 5. 1.3 

Acquire  TGTS  using  PLM 

Given  tactical  situation 

I AW  TACMAN 

2. 6. 5. 1.4 

Acquire  TGTS  using 

' 

Given  tactical  situation 

I AW  TACMAN 

2.6.5. 1.5 

Acquire  TGTS  using 
radar 

(rear  seat  acquisition) 

Given  tactical  situation 

I AW  TACMAN 

2.6.6 

Deliver  weapon 

2.6.6. 1 * 

Deliver  Aim  7 

Given  tactical  situation 

I AW  TACMAN 

2. 6. 6. 2 * 

Deliver  Aim  9 

Given  tactical  situation 

I AW  TACMAN 

2. 6. 6. 3 * 

Deliver  20mm  gun  fire 

Given  tactical  situation 

I AW  TACMAN 

Figure  C-l . Sample  Task  Listing  Page.  Entries 

Marked  with  an  Asterisk  (*).  were  the 
Superordinate  Objectives  for  Subsequent 
Analysis 

C 
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Figure  C-6.  Media  Selection  Diagram  for  One  of  the 
Subordinate  Objectives  in  Figure  C-2. 


Course,  Unit,  Lesson:  P.17.1 
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Figure  C-7.  Sample  Syllabus  Sheet.  Segments  1 through  8 
were  Grouped  from  the  Hierarchy  in  Figure  C-2 


Figure  C-8.  Sample  Syllabus  Sheet.  Segments  1 through  7 
were  Grouped  from  the  Hierarchy  in  Figure  C-2 
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APPENDIX  D 


TRAINING  SUPPORT  REQUIREMENTS  ANALYSIS 


This  appendix  contains  extracts  from  the  Training 
Support  Requirements  Report  (CDRL  Item  E0011).  Sections 
presented  pertain  to  cost  estimation  procedures  and  the 
figures  used  in  the  calculations. 
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TRAINING  SUPPORT  REQUIltt  Ml  NTS  ANALYSIS 


The  following  sections  of  this  appendix  contain  the  basic 
information  which  was  used  to  estimate  costs  for  development, 
implementation,  revision  and  maintenance  of  the  F-4  pilot  and 
RIO  courses.  Since  the  design  phase  was  under  contract,  when 
the  TSRA  was  prepared,  no  estimate  for  this  phase  was  calculated. 
Of  particular  relevance  are  the  development  costs  for  one  lesson 
in  each  of  the  seven  media  selected  for  use  in  the  F-4  training 
courses.  As  discussed  in  Section  III  of  this  report,  the 
number  of  lessons  per  medium  has  changed  as  a result  of  lesson 
specification  development,  which  began  after  the  original  TSRA 
was  produced.  Total  estimated  costs,  therefore,  have  changed, 
but  not  the  costs  per  lesson  in  each  medium. 

DEVELOPMENT  REQUIREMENTS 

The  development  phase  consists  of  the  authoring  and  pro- 
duction efforts.  Base  rate  values  for  these  efforts  were 
developed  in  the  format  specified  in  U0I-H-00040.  Table  0-1, 
Personnel  Hours  and  Materials  to  Develop  One  Lesson  of  In- 
struction, contains  the  basic  data  required.  Data  in  Table  D-l 
were  drawn  from  three  civilian  production  companies  and  from 
two  civil  service  (Navy)  organizations. 

In  Table  D-l  the  SME  has  been  considered  to  be  the  author. 

The  Instructional  Psychologist  has  been  considered  as  a part- 
time  program  manager  with  the  Instructional  Technologist 
providing  most  of  the  direct  support  and  assistance  in  the 
authoring  and  development  efforts.  The  production  manager  pro- 
vides the  liaison  between  the  graphics,  camera,  and  typing 
personnel.  In  addition,  he  will  provide  the  scheduling  of 
input  and  output  for  the  storyboard  and  illustration  requirements. 
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TABLE  D-l . BASIC  DATA  USED  TO  CALCULATE 
nEVFLOPMFNT  COSTS 
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The  personnel  hours  shown  in  Table  D-l  were  converted  to 
estimated  "loaded"  salaries  in  Tables  D-2  through  D-7.  The 
Approximate  Hourly  Rate  contains  the  estimated  additions  for 
overhead,  G&A  (General  and  Administrative)  and  profit  fees. 

Tables  D-2  through  D-7  show  the  estimated  costs,  by  media, 
for  production  of  the  lessons  in  the  T-4  pilot  and  RIO  syllabi. 

Materials  and  services  are  indicated  in  Tables  D-2  through 
D-7  using  dollar  amounts.  The  Development  Total  represents  the 
cost  for  one  reproducible  master  of  finished  lesson  materials 
for  the  given  medium.  Duplication  costs  were  not  estimated 
since  duplication  will  be  carried  out  by  the  government. 

A survey  of  available  training  assets  (equipment)  conducted 
during  Media  Selection  and  shown  in  Table  D-8,  indicates  that 
there  is  adequate  equipment  for  display  of  Videotape  and  adequate 
simulator/trainer  and  aircraft  availabilities.  However,  there  is 
a requirement  for  slide/tape  machines.  Additionally,  it  has  been 
determined  that  MCAS,  El  Toro  and  NAS,  North  Island  have  adequate 
Videotape  capabilities  as  well  as  reproduction  capabilities  that 
could  be  utilized  during  the  development  effort.  Further  research 
determined  that  Code  N-51,  Navy  Education  and  Training  Support 
Center,  Pacific  has  facilities  for  duplication  of  35mm  slides. 
Tasking  and  scheduling  would  be  required  through  Code  N-4,  CNET. 
Duplication  of  paper  materials  could  be  accomplished  through 
the  Navy  Publication  and  Printing  Service  and  tasked  through 
OPTAR.  No  specific  in-service  organization  appeared  to  have  a 
total  development  crew  that  could  be  made  available  for  full  F-4 
training  development.  It  is  recommended  that  the  production  of 
all  original  materials  be  contracted  and  that  duplication  of 
the  originals  be  done  by  the  government. 

The  equipment  required  for  development  are  those  items 
associated  with  a full  graphics  and  photography  capability  and 
include:  cameras,  lay-up  tables,  back-light  table,  photo-copiers, 
slide/tape  machine,  audio  pulsing  and  recording  machines,  etc. 


TABLE  D-2.  COSTS  FOR  PRODUCTION  OF 

MEDIATED  INTERACTIVE  LECTURES 
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TABLE  D- 3 . COSTS  FOR  PRODUCTION  OF  WORKBOOKS 
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MATERIALS/SERVICES  j 92.00 


TABLE  D-6.  COSTS  FOR  PRODUCTION  OF 

TRAINER/SIMULATOR  MATERIALS 


DEVELOP.  TOTAL  $453.60 


STS  FOR  PRODUCTION  OF  CHECKLISTS 
0 WORKSHEETS  FOR  FLIGHTS 


MATERIALS/SERVICES  i 115.00 


It  is  expected  that  the  contracted  development  company  will 
provide  the  required  equipment. 

Facility  requirements  for  development  would  include  sufficient 
space  for  camera  lay-up  (8 1 x 10'),  sound-proof  recording  (6'  x 6') 
and  for  graphic  equipment  (10'  x 20').  Attendant  tables,  desks 
and  chairs  would  also  be  required  if  on-site  production  was  in- 
dicated. The  facility  should  be  sufficiently  large  to  allow  for 
heat  dissipation  from  camera  lights  and  to  prevent  unnecessary 
noise  interference  when  recording. 

Facilities  currently  being  used  by  the  ISD  team  at  MCAS, 

Yuma  will  provide  sufficient  space  for  materials  production. 

These  include  two  12  x 10  offices  for  contractor  personnel  and 
two  11  x 11  offices  for  SME's.  Graphics  and  photographic 
activities  can  be  housed  in  a large  24  x 24  room.  In  addition, 
a 21  x 10  and  a 12  x 11  room  can  be  used  as  needed  by  production 
personnel.  All  rooms  are  well-lighted  and  have  sufficient 
electrical  power  to  support  production  efforts.  A sound-proof 
recording  facility  is  not  available  on-site.  This  facility  may 
be  supplied  by  the  narrator  if  he  is  a professional  with  access 
to  a recording  facility,  or  in-service  facilities  at  MCAS,  El  Toro 
and/or  NAS,  North  Island  may  be  used.  A dark  room  is  available 
at  MCAS,  Yuma.  Its  availability  for  contractor  use  could  not 
be  determined. 

IMPLEMENTATION  REQUIREMENTS. 

It  is  estimated  that  a maximum  of  30  instructors  will  be 
required  to  handle  a student  load  of  60  pilots  and  60  RIO's  per 
year  in  a FRS.  This  estimate  is  derived  from  a maximum  require- 
ment of  16  flights  per  day  with  a different  instructor  for  each 
flight,  and  2 instructors  per  day  for  each  of  the  three  simulators, 
the  learning  center,  and  the  Air  Combat  Maneuver  Range  (ACMR). 

This  totals  to  a maximum  of  26  instructors  available  and  allows 
approximately  15  percent  for  leave,  TAD,  etc.,  thus  yielding  30  in- 
structors assigned.  It  is  estimated  that  each  instructor  will 
be  available  four  to  six  hours  per  day. 
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Other  personnel  requirements  such  as  scheduling,  counseling* 
proctor/monitoring,  general  administration,  and  evaluation  can  be 
met  by  the  30  assigned  instructors.  In  addition,  a minimum  of 
two  enlisted  personnel  will  be  required  to  distribute  materials, 
maintain  student  records,  and  service  learning  center  equipment. 

The  primary  equipment  required  for  implementation  at  each 
FRS  will  be  audio-visual  carrels  in  which  the  slide-tape  lessons 
will  be  presented.  A minimum  of  eight  carrels  will  be  required 
at  each  FRS.  Existing  videotape  equipment  at  the  three  FRS's 
can  be  used  for  the  videotape  lessons. 

Each  operational  and  reserve  squadron  provided  with  course 
materials  will  require  one  functioning  audio-visual  carrel  plus 
one  spare.  If  feasible  a complete,  centralized  learning  center 
may  be  set  up  at  locations  having  more  than  one  squadron. 

Equipment  costs  to  set  up  learning  centers  at  the  three 
FRS's  are  summarized  in  Table  D-9.  The  current  costs  are  based 
on  GSA  prices;  the  projected  costs  include  a 10  percent  inflation  factor 
to  account  for  price  increases  prior  to  equipment  purchase.  The 
quantity  of  ST  carrels  required  is  eight  per  FRS  with  two 
additional  cassette  players  and  projectors  per  FRS  included  as 
on-line  spares. 

It  is  recommended  that  two  complete  ST  carrels  be  procured 
for  each  user  squadron  rather  than  one  complete  carrel  plus 
on-line  spares.  One  operational  carrel  will  supply  the  training 
needs  of  each  squadron. 

The  learning  center  at  each  FRS  should  contain  approximately 
350-400  square  feet  to  accommodate  the  carrels,  an  instructor 
station,  and  to  allow  free  movement  around  the  carrels.  A separate, 
but  preferably  adjoining  room  will  house  the  lesson  materials  and 
will  be  used  for  storage  and  check  in  and  out  of  materials.  This 
room  should  be  secure  to  allow  for  storage  of  classified  materials. 

It  should  be  a minimum  of  100  square  feet  and  contain  200  feet 
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of  shelf  space  for  materials'  storage.  A classroom  capable  of 
seating  10-15  students  will  be  needed  for  lectures  and  periodic 
class  meetings.  Approximately  eight  briefing  rooms  will  be  required 
for  flight  briefings.  Space  lor  briefings  prior  to  simulator 
lessons  will  also  be  required. 

Facilities  requirements  at  other  squadrons  will  be  minimum 
since  there  will  be  no  more  than  two  carrels  and  storage  space 
for  one  set  of  materials  per  squadron.  Approximately  150  square 
feet  will  be  required.  Materials  can  be  stored  in  a secure 
cabinet  in  this  space. 

A preliminary  survey  to  determine  the  availability  of 
facilities  at  the  three  FRS's  indicated  that  the  required  space 
is  available.  Specific  rooms  to  be  used  for  the  learning  centers 
could  not  be  identified.  Based  on  the  general  characteristics  of 
rooms  that  may  be  converted  to  learning  centers,  it  is  estimated 
that  the  following  modifications  will  be  required:  (1)  installation 
of  carpeting  in  the  350-400  square  foot  learning  center:  (2) 
additional  electrical  power  to  provide  the  4000-4500  watts  required 
when  all  carrels  are  in  use:  (3)  installation  of  shelves  or  the 
procurement  of  cabinets  to  provide  materials  storage  space. 

Classroom  and  briefing  facilities  were  judged  to  be  sufficient  at 
all  FRSs . 

FIVE  YEAR  EVALUATION  AND  MAINTENANCE 

This  phase  of  the  ISD  effort  is  concerned  with  program  re- 
vision and  maintenance  requirements.  After  the  training  program 
has  been  implemented  and  evaluated,  there  will  be  first  year 
revision  requirements.  During  the  first  year,  many  revisions  will 
occur  and  be  integrated  into  the  training  program.  This  requires 
more  support  personnel  during  the  first  year  of  a five  year 
evaluation  and  maintenance  effort  than  for  the  last  four  years. 

Total  requirements  for  first  year  revisions  and  quality 
control  activities  were  determined  by  multiplying  the  personnel, 
materials,  and  services  directly  tied  to  courseware  development 


152 


NAVTRAEQUIPCEN  77-C-0081-1 


4> 

and  production  by  .35.  Annual  requirements  for  the  next  four 
years  were  determined  by  applying  a factor  of  .20. 

RECOMMENDATIONS 

It  is  recognized  that  the  TSRA  results  have  indicated  a 
costly  development  program  for  the  F-4  Pilot  and  RIO  courses. 

The  estimated  manhour  requirements  are  believed  to  be  very  accurate 
for  those  personnel  shown.  The  dollar  amounts  approximated  for 
the  development  personnel  are  those  that  a normal,  well-skilled 
professional  organization  would  charge.  However,  not  all 
organizations  utilize  the  same  overhead  rate,  nor  do  all  organi- 
zations have  personnel  skilled  in  segregated  professions.  That 
is,  the  Instructional  Technologist  may  also  be  the  Program 
Manager,  the  Production  Manager  may  also  be  the  proof  reader,  and 
the  paste-up  artist  may  be  the  graphics  specialist  or  photographer. 
The  most  obvious  cost  area  is  the  graphic  specialist  and  photo- 
grapher in  the  slide- tape  lesson  area;  however,  they  are  required 
specialists.  It  is  suggested  that,  in  other  areas,  personnel  who 
are  multi-talented  be  considered  acceptable  specialists  and  that 
the  delineation  of  personnel  categories  by  title  not  be  utilized 
in  the  acquisition  of  a development  team.  In  some  cases,  a SME 
could  be  utilized  for  production  management  or  for  paste-up  work. 
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